Ingestion Hardening

Current Status

Indexed connector ingestion hardening is complete for connectors that declare indexed sync in the runtime manifest. Live-query pagination, tombstone retention, and malware/quarantine remain separate deferred tracks.

Implemented Indexed Hardening

• Indexed taps emit explicit sourceType values; API rejects indexed file payloads that omit source semantics.
• Pagination, provider delta handling, stale cursor fallback, delete propagation, and Retry-After rate-limit backoff are covered by shared tap tests for manifest-backed indexed providers.
• The embedding path normalizes indexed text before chunking so raw HTML tags, scripts, tracking boilerplate, and empty boilerplate do not pollute retrieval while useful headings, list/table text, mail context, and file/source context are preserved.
• Unchanged content is not re-embedded. Access-only, connector-binding-only, and metadata-only changes update MongoDB/Qdrant payload metadata through reconciliation.
• Qdrant connector collection setup creates payload indexes for metadata.permissions, metadata.connectorIds, and metadata.contentId before vector upsert, filtered delete, or payload reconciliation.
• Provider deletes, full-snapshot stale cleanup, disconnect cleanup, resource-scope rebuild cleanup, and indexed-mode-off cleanup use the shared indexed-artifact reconciliation policy.
• Connector sync queue dispatch is user/workload-profile fair by default; provider-specific caps remain operator overrides.
• Manual sync accepts mode: "incremental" | "reconcile". Reconcile uses full-refresh behavior and the same queue fairness path.
• Real-provider indexed E2E covers current manifest-backed indexed providers through connect, sync, ready, disconnect, and cleanup.

Deferred Scope

Live-query pagination is intentionally out of scope for this hardening track. Tombstone retention and operator purge workflows remain deferred; hard-delete parity now runs through shared cleanup/reconciliation seams so tombstones can be added later without rewriting indexed ingestion. Malware scanning or quarantine remains deferred until provider, lifecycle, failure UX, and operator review semantics are accepted.

Local Reset Guidance

Prefer ephemeral test services for verification. Do not clear chats for indexed ingestion work. If local connector indexed data must be reset after a metadata contract change, tell the operator first and name the exact Qdrant/Mongo target before deleting anything; use connector disconnect/reconcile flows where possible instead of ad hoc database mutation.

Triage Before Implementation

The retired root backlog mixed completed work, stale notes, deferred ideas, and real future work. Do not implement items from that old list directly. Re-check current code and canonical feature specs first, then promote only still-valid work into the owning feature spec/story/test set.

Potential non-current-PR candidate areas to promote into owning specs when accepted:

• connector/API input validation coverage and OpenAPI documentation
• production observability, metrics, tracing, dashboards, and backup/restore runbooks
• deployment-to-deployment service authentication hardening, including request signing or mTLS where appropriate
• OAuth scope lifecycle UX, audit logging, provider revocation, and reauthorization behavior
• provider-backed upload destination defaults, upload readiness notifications, and future version/re-upload metadata
• principal namespace design for future non-owner permission entries
• provider quota documentation and chunking strategy beyond the current embedding policy
• CI/CD gates for unit, integration, E2E, release-candidate, and staging deployment lanes

Canonical Sources

• docs/features/ingestion/spec.md
• docs/features/connectors/spec.md
• apps/api/src/connectors
• apps/api/src/ingestion
• packages/types/embedding-policy.json