BullMQ

BullMQ is the Redis-backed queue library used by the API connector sync runtime.

What It Owns

Connector sync enqueue, worker, retry, restart, and stale-run recovery behavior in apps/api.
Queue-level coordination over Redis.
Runtime failure modes around worker restarts, provider sync errors, and status publishing.

Architecture Decision

Decision: keep queue mechanics in BullMQ and keep product policy in API connector orchestration code.

BullMQ provides queue primitives; it should not become the place where connector authorization, customer policy, or user-facing recovery copy is decided.

The current API runtime uses the connector-sync-jobs queue for connector sync work. Worker concurrency comes from DATA_SOURCE_SYNC_QUEUE_CONCURRENCY and defaults to 2. Default dispatch fairness is connector-agnostic: one user cannot consume all worker slots, indexed full-refresh/rebuild jobs have their own bounded profile cap, and BullMQ priorities keep connect/manual/incremental work ahead of heavier reconciliation work. Provider caps from CONNECTOR_SYNC_PROVIDER_CONCURRENCY_CAPS are optional operator overrides, not the primary fairness model. Disconnect cleanup currently runs through the API in-process cleanup queue, and embedding ingestion starts from validated sync events rather than a separate BullMQ ingestion queue.

How To Upgrade

Check BullMQ release notes and Redis compatibility guidance.
Upgrade the API dependency:
Terminal window
```
# from apps/api
bun update bullmq
```
Keep Redis image compatibility in view. If BullMQ changes Redis requirements, update Redis at the same time.

Run connector queue and status lanes:

bun run test:api:connectors:e2e:ephemeral
bun run test:api:connectors:restart:e2e
bun run test:api:connectors:realtime

Current Limits

Queue behavior must fail closed around auth and connector policy.
Do not encode user-visible English in queue payloads. Use status/error/detail codes at web-facing boundaries.
Do not rely on prompt text or semantic keyword matching for queue routing decisions.

Canonical Sources

apps/api/src/connectors/syncTriggerRunner.ts
apps/api/src/connectors/
apps/api/package.json
docker-compose.yml
docs/features/connectors/spec.md
docs/features/ingestion/spec.md