Skip to content
Leverage AI Developer Support

Verification + Evals

Verification depends on the surface you touched. Keep the run focused, but do not skip type/build checks for changed apps.

Baseline Support-Dev Checks

Section titled “Baseline Support-Dev Checks”
Terminal window
bun run --cwd apps/support-dev check-types
bun run --cwd apps/support-dev build

Common Verification Lanes

Section titled “Common Verification Lanes”
Surface Useful checks
Support-dev content bun run --cwd apps/support-dev check-types, bun run --cwd apps/support-dev build
Web UI App-specific type/test/build commands, plus browser or visual verification when visible behavior changes
API / Mastra Type/test commands plus chat eval lanes when orchestration, tools, memory, scorers, or HITL behavior changes
Connectors Unit/integration tests, seed impact check, provider runtime smoke when configured, and relevant web/API contract checks
Auth / SSO Auth route tests, browser auth checks, and provider-specific setup verification
CMS App build/type checks, Payload route/collection tests, and seed/admin workflow checks
Coverage bun run test:coverage or bun run test:pr:coverage for the product coverage gate

Coverage Gate

Section titled “Coverage Gate”

Coverage work must add meaningful regression protection, not line-touching tests. Do not add import-only, snapshot-only, or mock-call-only tests merely to raise coverage. New tests added for coverage must exercise real intended runtime behavior, meaningful edge cases, or fail-closed/error paths. For auth, security, persistence, connector, customer-boundary, and data-integrity surfaces, tests must assert durable or user-visible outcomes directly. If a coverage target is low, first identify the highest-risk uncovered behavior and cover that behavior before adjusting thresholds or exclusions.

The product coverage gate currently enforces lines 80% and functions 80% for every configured target; branches and statements are reported until stable across all runners. Use the coverage report to find high-risk uncovered behavior, then add focused tests that prove real contracts before changing thresholds or exclusions.

Browser Test Runner Notes

Section titled “Browser Test Runner Notes”

Customer CMS Playwright lanes run the Payload/Next dev server with next dev --webpack. Keep that documented flag for local and CI browser tests unless a replacement is verified in bun run test:pr:e2e; the default Turbopack dev path has produced CI-only Payload external-module resolution failures and Turbopack panics.

Manual Browser Smoke Checks

Section titled “Manual Browser Smoke Checks”

Feature-owned manual smoke checks live with the support-dev page that explains the surface:

  • Chat Runtime covers private chat, focused chat, and guardrail smoke checks.
  • Chat Message Lifecycle covers branch editing, branch switching, branch-scoped delete, and reload persistence.
  • Human In The Loop covers chat-thread-scoped HITL information-request, refresh, cancel, and submit flows.
  • Response Templates covers template authoring, chat selection, thread persistence, and policy-boundary checks.

For QA release validation against a deployed environment, use Feature QA Smoke Map. That page avoids local terminal commands, mocked fixtures, and developer-only prompt packs.

Developer Connector Verification

Section titled “Developer Connector Verification”
  • Local Connector Prompt Packs covers gitignored prompt-pack generation and browser-backed checks against developer-local connector accounts.

Mastra Eval Entry Points

Section titled “Mastra Eval Entry Points”
  • Deterministic chat and workflow checks are summarized in Chat Evals And Regression Testing.
  • Local connector prompt packs are summarized in Local Connector Prompt Packs. Use them as developer-local evidence for real connected accounts, not as a substitute for deterministic gates.
  • The Chat Orchestration Evals GitHub workflow runs automatically for chat-related PRs and dev pushes, runs on a weekday schedule, and remains manually dispatchable for release-candidate refs.
  • Scheduled Chat Orchestration Evals failures open or update a labeled GitHub issue so provider, model, or configuration drift is visible outside the Actions list; a later scheduled recovery closes that issue.
  • Main Release Health / chat-scheduled-health blocks PRs to main while any issue labeled showstopper and chat-orchestration-evals is open. Configure that job as a required status check in the main branch protection or ruleset.
  • Persisted experiment runs populate Mastra datasets and experiments when the experiment lane is executed by the workflow or local review command.
  • Studio can inspect registered agents, workflows, canonical live tools, scorers, datasets, and traces.

Canonical Sources

Section titled “Canonical Sources”
  • apps/support-dev/package.json
  • package.json
  • apps/api/package.json
  • apps/api/src/mastra/evals
  • apps/api/src/mastra/scorers
  • docs/features/chat/spec.md
  • docs/features/connectors/spec.md