Local Connector Prompt Packs

Use this developer-verification lane when a change could affect connector-backed chat answers, source routing, HITL information requests, broad workplace prompts, or chat answer quality with a developer’s local connector accounts.

Local connector prompt packs complement deterministic Mastra evals. They are not a replacement for test:local, mocked eval fixtures, or focused API/web regression tests.

What This Covers

• Generate a local prompt pack from the currently logged-in web user’s chat-ready connectors.
• Run those prompts through real local private chats and the production chat route.
• Capture answer text, sources, HITL outcomes, and basic quality flags in gitignored artifacts.
• Use the generated Markdown pack as a manual agent-browser checklist when you need visual review.
• Run follow-up cases with generic same-thread setup prompts so stale source/query continuation bugs are visible in the saved chat transcript.
• Exercise ordinary connector prompts in all-scope private chat by default so source-selection and non-focus orchestration bugs are visible. Focused connector, connector-account, library, and mixed-resource cases should be explicit smoke cases, not the default for every connector prompt.

Developer Boundary

This workflow assumes a local repo checkout, dev servers, an authenticated agent-browser session, and developer-local connector data. It belongs to developer verification, not QA release validation; QA checks should target staged releases and must not require local connector prompt packs or terminal commands.

Prerequisites

1. Start the local stack.
2. Sign in to the web app with the standard agent-browser login command:

bun run agent-browser:login:web

3. Add representative local connectors through the product UI.

The pack is only as strong as the local connector data available to that developer. Thin connector sets should be reported as thin coverage, not treated as broad orchestration proof.

Generate A Pack

bun run chat:prompt-pack:local -- --force

This writes:

• tmp/connector-chat-prompts/latest.json
• tmp/connector-chat-prompts/latest.md

The tmp/connector-chat-prompts/ directory is gitignored. Do not commit generated prompt packs, raw answers, provider facts, screenshots, or run artifacts.

Run The Pack

bun run chat:prompt-pack:run:local -- --kind broad --all --auto-submit-hitl --timeout-ms 180000

Useful variants:

• --dry-run previews the cases without creating chats.
• --connector <key>, --domain <domain>, --kind <kind>, and --limit <n> bound the run.
• Use --domain code-repository to exercise repository/code/commit prompts when GitHub Repositories is configured.
• Use --domain business-record to exercise business-record prompts when Odoo or SharePoint lists are configured.
• Use --domain knowledge-page to exercise Confluence, SharePoint, or OneNote knowledge-page prompts when those connectors are configured.
• --include-follow-ups includes prompt cases that first send their setup prompts in the same chat, then send the follow-up prompt under test.
• --omit-answers skips raw answer capture for privacy-sensitive or CI-shaped runs.
• By default, created chats remain in the web app so a human can inspect the answer, Sources panel, HITL state, and thread transcript.
• --delete-chats cleans up chats created by the run. Use it for CI-style or privacy-sensitive cleanup, not when the goal is manual answer-quality review.

Run artifacts are written under tmp/connector-chat-prompts/runs/. Review the JSON for passedBasicChecks, source coverage, unresolved HITL placeholders, and answer text when answers are captured.

GitHub connector examples:

bun run chat:prompt-pack:run:local -- --connector github-repositories --limit <n>
bun run chat:prompt-pack:run:local -- --connector github-issues --limit <n>
bun run chat:prompt-pack:run:local -- --connector github-pull-requests --limit <n>
bun run chat:prompt-pack:run:local -- --domain code-repository --limit <n>

Odoo/business-record examples:

bun run chat:prompt-pack:run:local -- --connector odoo --domain business-record --limit <n>
bun run chat:prompt-pack:run:local -- --domain business-record --include-follow-ups --limit <n>

Confluence/knowledge-page examples:

bun run chat:prompt-pack:run:local -- --connector confluence --domain knowledge-page --include-follow-ups --limit <n>
bun run chat:prompt-pack:run:local -- --domain knowledge-page --include-follow-ups --limit <n>

Microsoft knowledge-source examples:

bun run chat:prompt-pack:run:local -- --connector sharepoint --domain knowledge-page --include-follow-ups --limit <n>
bun run chat:prompt-pack:run:local -- --connector sharepoint --domain file-resource --limit <n>
bun run chat:prompt-pack:run:local -- --connector sharepoint --domain business-record --limit <n>
bun run chat:prompt-pack:run:local -- --connector onenote --domain knowledge-page --include-follow-ups --limit <n>

CI Status

Today this is a developer-local lane, not a default GitHub CI gate. The script can run in a CI-shaped mode, but a real CI job should be added only as a protected integration or nightly lane with dedicated fixture connector accounts, protected provider secrets, --omit-answers, --delete-chats, and an explicit artifact-retention policy. Do not upload raw provider facts, raw answers, screenshots, HAR files, chat ids, connector ids, or generated prompt packs from CI artifacts.

Manual Browser Review

Use tmp/connector-chat-prompts/latest.md as the human-readable checklist for agent-browser or manual UI testing. Broad, cross-reference, discovery, fact-seeding, positive, negative, and general-knowledge cases should all be sampled when connector orchestration quality is under review.

Manual and automated runs are intentionally similar: both should use natural prompts against real local chat execution. Manual review adds visual checks for source drawers, HITL slideovers, reasoning/activity UI, sidekick state, and answer trustworthiness. Do not pass --delete-chats during manual review unless the reviewer explicitly does not need the chats to remain visible.

Default generated connector cases use Focus: all. Use focus-specific cases only when verifying a focus boundary, such as connector-account focus, Library focus, or mixed file/folder focus. If an all-scope prompt does not show an expected connector as checked in Sources, treat that as an orchestration/source-selection regression even when the same prompt works in focus mode.

How To Interpret Results

• Passing the local pack proves the current local connector/account mix did not show the checked failure modes.
• Failing answers should be triaged like real orchestration regressions: inspect sources, HITL state, logs, and eval coverage before changing prompts or scoring.
• General-knowledge control prompts should answer without fabricating connector sources.
• Source-backed broad prompts should either cite/check relevant connector domains or clearly say the requested evidence was not found.
• Code-repository positive checks should preserve repository full name, file path, ref or commit details when available, language, and structured source actions without printing GitHub URLs in answer prose. They should also say when provider search was bounded, incomplete, or not semantic whole-codebase recall.
• GitHub work-item positive checks should sample both object prompts and facet prompts. Labels and milestones need separate available/configured inventory checks plus filtered issue/PR usage checks, because inventory evidence alone does not prove an issue or PR currently uses that facet.
• Business-record positive checks should sample model/list inventory, field/column inventory, record search/list/detail, custom fields, relationships, bounded aggregates when supported, and source switching. Odoo checks should disclose inaccessible or unavailable models instead of implying complete coverage. SharePoint checks should disclose selected-site/list boundaries and unavailable lists or unsupported tenant-specific Search schema fields.
• Knowledge-page positive checks should sample Confluence spaces, SharePoint sites, OneNote notebooks, title/text page search, pages in a container, page details/snippets, recent pages, hierarchy, provider facets, and source/provenance follow-ups. Confluence checks should disclose bounded provider search, company-managed service/bot provenance when applicable, unsupported attachment content extraction, unsupported writes, and Cloud Basic auth being unsupported. SharePoint and OneNote checks should disclose selected-resource boundaries, unsupported writes/indexed recall, unsupported media or attachment content, and bounded provider search where relevant.
• Opportunistic checks cover tenant-dependent facets that may be absent from a developer’s real account, such as optional labels, comments, tree children, attachment metadata, or child records. They still fail generic no-source answers, but a scoped no-match without sources is an acceptable local outcome. Required positive coverage for those facets belongs in deterministic eval fixtures with neutral provider-shaped data.
• Negative/no-match checks fail basic scoring when the answer invents a result or omits no-match, unsupported, or unavailable wording. Treat that as an answer-quality regression, not just a prompt-run warning.
• Negative/no-match markers are generated into ignored prompt-pack artifacts. Do not commit exact marker literals into tests, docs, or generator source, because repository-backed connectors may find them in the repo and convert a negative prompt into a real source-backed match.
• File-resource positive checks include a mixed live file+folder focus case when local connectors expose browsable files and folders. The runner expects selected live refs to work without canonicalContentId, and it flags answers that only describe file names when the prompt needs file content.
• SharePoint file-resource checks should distinguish document-library files/folders from OneDrive files when same-title decoys exist, and should report unsupported or binary extraction as bounded coverage rather than inventing file content.
• Follow-up checks run in the same thread as their setup prompts. Source-backed follow-ups must still attach source results, stay on the intended connector/domain, and fail basic scoring if they answer generically without checked sources.
• File-resource follow-up checks include a PDF-listing setup followed by a terse What else do you see? prompt. The expected behavior is to stay on the prior file connector, drop stale PDF filtering, include owned/shared top-level coverage when available, and phrase the result as bounded rather than complete inventory.
• Business-record follow-up checks should keep the prior model/customer/project context only when the follow-up is referential, and should switch sources when the user explicitly asks to check another connected source.
• Basic scoring also fails user-visible internal no-evidence wording and hidden system files such as .DS_Store; these checks are diagnostics and must not become production routing logic.
• Private provider facts discovered by a pack can inspire deterministic evals only after being rewritten into neutral synthetic fixtures with positive and negative decoys.

Canonical Sources

• .agents/skills/connector-chat-verification/SKILL.md
• scripts/generateLocalConnectorPromptPack.ts
• scripts/runLocalConnectorPromptPack.ts
• apps/api/src/mastra/evals
• docs/features/chat/spec.md
• docs/features/connectors/spec.md