Meltano

Meltano powers the indexed connector ingestion path for providers that retain searchable indexed data.

What It Owns

apps/meltano owns the Meltano project, plugin config, local runtime bootstrap, and project-managed Python runtime.
apps/api queues and invokes connector sync work that can run through Meltano.
packages/taps/* holds repo-owned Singer taps used by the Meltano runtime.
Docker local development can run meltano/meltano:v4.2.0 through the meltano compose profile.

Architecture Decision

Decision: use Meltano/Singer extraction for indexed connector ingestion before adding custom one-off ingestion runners.

This keeps provider extraction observable, testable, and compatible with data-platform conventions while the API remains the orchestration, policy, embedding, and sync-status boundary.

Meltano reports sync runtime facts through API /sync-events. The API owns sync status, embedding, metadata/vector persistence, and user-facing realtime updates; Meltano does not trigger ingestion through MongoDB change streams.

How To Run Locally

Normal local connector work should use the project-managed local runtime:

bun run --cwd apps/meltano setup:local-runtime
bun run test:meltano

Keep MELTANO_RUNNER=local for routine API-driven sync debugging. bun run docker:up starts shared infrastructure only; use bun run docker:up:meltano only when you need the optional Docker Meltano runner. If you switch runner modes, restart the API process so sync workers pick up the new setting.

How To Upgrade

The repo-managed Meltano CLI is pinned in two owned runtime paths:

apps/meltano/scripts/bootstrapLocalRuntime.ts: local project-managed CLI install through MELTANO_VERSION
docker-compose.yml: Docker runner image on leverage-ai-meltano

Do not treat this as a package.json dependency upgrade unless the repo adds a real Node dependency. Before changing the pins, check PyPI for the latest stable meltano release and confirm the matching meltano/meltano:v<version> Docker tag is active. Keep MELTANO_CLI_DISABLE_VERSION_CHECK=true in runner/bootstrap/Docker env paths so Meltano advisory stderr cannot appear in connector sync logs or user-facing sync failures.

Rebuild and verify local runtime:

bun run --cwd apps/meltano setup:local-runtime
apps/meltano/venv/bin/meltano --version
bun run test:meltano

Rebuild and verify Docker runtime:

docker compose --profile meltano pull leverage-ai-meltano
bun run docker:down
bun run docker:up:meltano
docker compose exec leverage-ai-meltano sh -lc 'cd /project && /venv/bin/meltano --version && /venv/bin/meltano install'

Connector sync smoke coverage should pass before merging a Meltano runtime upgrade:

bun run test:api:connectors:e2e:ephemeral
bun run test:api:connectors:restart:e2e
bun run test:api:connectors:real-provider:doctor
bun run test:api:connectors:real-provider:e2e:all

Current Limits

Slack and Jira are live-only in the current repo shape. Do not add indexed Meltano paths for them without a canonical feature decision.
Indexed connector mode is not zero-persist: API ingestion stores extracted chunk text and vectors for retrieval. Live-only mode is the user-facing path that avoids embedding sync and indexed artifact retention where the connector exposes both modes.
The Docker runner stages read-only repo source into writable /project so Python plugin installs can build local taps without writing generated metadata into the repo checkout.

Canonical Sources

apps/meltano/meltano.yml
apps/meltano/scripts/bootstrapLocalRuntime.ts
apps/api/src/connectors/syncTriggerRunner.ts
packages/taps/
docker-compose.yml
docs/features/connectors/spec.md
docs/features/ingestion/spec.md