Indexed Ingestion

Use this page when you need the high-level shape of connector embeddings, token refresh during sync, deduplication, and indexed artifact cleanup.

What Indexed Mode Stores

indexed mode stores searchable retrieval artifacts inside the platform. Eligible provider content is extracted, normalized, chunked, embedded, and stored as:

• MongoDB connector metadata, canonical content records, access bindings, checksums, skip/failure state, and indexed readiness state.
• Qdrant vectors with extracted chunk text and permission/provenance payloads.

The chunk text is stored intentionally. Mastra/Qdrant retrieval needs text snippets for grounded answers; fetching every matching source item back from Google, Microsoft, or another provider during each chat turn would add provider latency, rate-limit exposure, token-refresh dependency, and repeated extraction work.

For providers with both live and indexed support, enabledModes: ['live'] is the user opt-out from embedding sync and indexed artifact retention. enabledModes: ['live', 'indexed'] opts into extracted chunk text plus vector storage for eligible content.

Token Refresh During Sync

Connector sync workers and Meltano taps do not refresh provider tokens themselves.

1. The worker/tap calls the service-authenticated API token-context route with only { connectorId }.
2. API resolves the connector owner, linked account, and provider context server-side.
3. API calls the auth service internal OAuth token route.
4. Better Auth returns a usable provider access token and refreshes expired access tokens where the provider grant permits it.
5. If refresh or scope validation cannot recover, the connector moves toward a user-safe reauthorize/reconnect state instead of letting the worker guess.

This keeps OAuth refresh authority in apps/auth and keeps sync workers from choosing arbitrary users, accounts, or provider tokens.

Ingestion Flow

1. Provider sync emits validated sync facts to API through /sync-events.
2. API handles sync-completed file batches through the canonical embedding ingestion service.
3. Unsupported, oversized, or hard-blocked files are skipped according to ingestion policy.
4. Supported content is normalized, chunked, embedded, and written to Qdrant with chunk text and authorization metadata.
5. Connector status, synced counts, indexed counts, and recovery details flow back through the connector status/realtime contracts.
6. Provider-backed uploads queue connector sync work after provider write success so the ingestion path can observe the new provider file.
7. If the indexed/skipped metadata belongs to a provider-backed Library upload, API reconciles the matching upload session into a Library job state and emits library-job-updated after persistence.
8. Operators or users may request manual sync with mode: "incremental" or mode: "reconcile". Reconcile uses full-refresh behavior and the same fair queue dispatch path as other indexed sync work.

MongoDB remains authoritative for metadata, permissions, and access bindings. Qdrant is the retrieval index, not the permission authority. Before the first vector upsert, filtered vector delete, or vector payload reconciliation, the connector vector store creates keyword payload indexes for metadata.permissions, metadata.connectorIds, and metadata.contentId so filtered retrieval and cleanup do not depend on an unindexed payload scan.

On connector dashboard rows, Synced means provider items the sync observed and reconciled into connector metadata. Indexed means items that became searchable after eligibility checks, extraction, normalization, permission reconciliation, and embedding. These counts can differ for skipped files, deleted or unshared content, in-progress embedding, or items that are supported for live access but not indexed storage. Live-only mode hides these indexed-sync counters because it intentionally avoids connector-owned indexed artifacts.

Library Upload Readiness

Provider-backed uploads use the same connector ingestion path as other indexed file content. The important difference is user-visible readiness: people wait on the inline Library contents status row and sometimes on a focused-chat Library scope.

• LibraryJob is the provider-upload job record. Upload stores upload-session/provider metadata such as destination folder, provider file id, and linked Library item id.
• Confirmed same-name upload overwrites store the provider file id being replaced on the Upload record so readiness reconciliation can update the existing Library item instead of creating a duplicate item when the replacement indexes to a new content id.
• Provider-backed uploads request and queue connector sync after provider write success before reporting sync-requested.
• Embedding ingestion writes durable file metadata/vector state first.
• API reconciles pending uploads for the connector into ready, actionable not-indexed, failed, or non-actionable terminal Library job snapshots.
• Connector settings changes that toggle indexed mode proactively run that same provider-upload readiness reconciliation so open Library pages receive Library job updates without waiting for a later sync or snapshot.
• Turning indexed mode off deletes connector-owned indexed/vector artifacts only; it preserves Library items, Upload records, and provider file metadata.
• Upload-capable connectors without indexed ingestion enabled are not valid Library upload destinations. If an existing provider-backed upload already landed there, API reconciles it terminally rather than waiting indefinitely; Library readiness and inline Library contents status rows suppress that state because indexed ingestion was intentionally left off.
• The web app receives library-job-updated and refreshes Library job snapshots with compact Library readiness on reconnect.
• Import, delete, and reindex Library job producers use the same LibraryJob realtime path. Parser/extraction progress and batch maintenance Library job producers remain planned.

Deduplication And Access Bindings

When a provider exposes a stable native id, ingestion stores one canonical content record per provider + referenceId. If multiple users or connectors can see that same provider item, the platform records access bindings rather than duplicating canonical content and vectors per user.

Retrieval remains permission-scoped:

• the vector query uses user/connector metadata filters before search,
• API verifies returned results against MongoDB access bindings before grounding text or citations are emitted,
• permission revocations remove access bindings so future retrieval no longer returns the content for that principal.
• access-only or connector-binding-only changes update Qdrant payload metadata without re-embedding unchanged content.
• active connector ids, permission scope, source type, and access-scope hash are reconciled through one shared indexed-artifact policy instead of provider-local Qdrant update code.

Updates And Deletes

• Updated content replaces the previous canonical indexed chunks after change detection sees a new provider ETag, size, modified time, or checksum.
• Deleted provider items hard-delete or reconcile metadata, access bindings, and Qdrant points according to the shared indexed-artifact cleanup policy. Qdrant deletes include expected connector and permission scope filters when binding data is available. Tombstone retention remains a deferred operator-retention feature.
• Switching a user-selectable connector back to enabledModes: ['live'] stops future embedding sync and deletes connector-owned indexed artifacts without deleting app-owned Library/upload records.
• Disconnect cleanup schedules the same artifact cleanup path so stale vectors do not remain queryable after the connector is removed.
• Provider full snapshots, delta delete events, resource-scope reductions, and indexed-mode-off cleanup all enter the same reconciliation seam before Mongo metadata is removed.

Queue Fairness

Connector sync dispatch uses queue priorities plus active-dispatch caps by user and workload profile. Indexed rebuild/reconcile work cannot starve smaller incremental jobs, and one user cannot consume the whole worker pool. Provider-specific concurrency caps remain available as operator overrides for provider quota protection, but they are not the primary fairness model.

Local Data Reset Guidance

Use ephemeral services for tests and local verification when possible. Do not run chat:data:clear for connector ingestion work. If a local metadata contract change leaves stale connector vectors, prefer reconnect/disconnect or manual reconcile first; if a Qdrant or Mongo reset is still needed, tell the operator the exact collection/database target and impact before deleting anything.

Current Limits

• Indexed mode is a data-retention choice. Do not describe it as zero-persist provider content; live-only mode is the no-indexed-artifact path for providers that expose that option.
• Live-query pagination is not part of indexed ingestion hardening.
• Future tombstone retention should extend the shared cleanup/reconciliation policy rather than introducing separate provider-specific delete paths.

Canonical Sources

• docs/features/ingestion/spec.md
• docs/features/connectors/spec.md
• docs/features/connectors/decisions/ADR-010-connector-mode-model.md
• apps/api/src/ingestion/connectorEmbeddingIngestionService.ts
• apps/api/src/ingestion/qdrantUpsertPayload.ts
• apps/api/src/uploads/providerUploadService.ts
• packages/contracts/src/realtime/events.ts
• apps/api/src/connectors/oauthTokenClient.ts
• apps/auth/src/index.ts
• apps/meltano/src/services/api_client.py
• apps/support-dev/src/content/docs/dependencies/qdrant.md

Official Docs

• Mastra vector query tool
• Mastra Qdrant vector store
• Qdrant payloads
• Qdrant indexing