Chat Evals And Regression Testing

The chat orchestration eval system is documented canonically in docs/features/chat/spec.md. Use this page for operator-facing workflow and regression triage, not as a second source of product truth.

Canonical Sources

• docs/features/chat/spec.md owns the durable eval behavior, thresholds, and release policy.
• docs/features/chat/stories/chat-orchestration-implementation.stories.md tracks implementation progress and change history.
• apps/api/src/mastra/evals contains the executable scenario catalog, fixtures, and dataset builders.
• apps/api/src/mastra/scorers contains the shared scorer registration and release gate logic.
• apps/support-dev/src/content/docs/ mirrors the operator-facing workflow.

What The Eval System Covers

• Orchestration regressions in the private and public chat paths.
• Wrong tool, agent, or execution-path selection.
• Weak, off-base, or under-grounded answers.
• Focus-mode, HITL, response-template, and skill regressions.
• Grounding and citation regressions, including future segment-level grounding checks.

How To Run It

Blocking local and PR gate

• bun run test:local is the canonical blocking local suite.
• bun run test:pr currently points at the same suite when you want the command name to match PR/merge discussions.
• bun run test:local includes:
  • bun run test:local:base
  • bun run test:api:chat:evals
  • bun run test:api:chat:hitl
  • bun run test:web:chat:hitl
  • bun run test:api:chat:runtime-smoke
  • bun run test:web:e2e
  • bun run --cwd apps/cms-customer test:e2e

Faster base local lane

• bun run test:local:base runs the repo suites, local CMS integration, and guarded connector E2E without the chat/browser blocking lanes.

Chat-specific commands

• Repo root, deterministic chat evals: bun run test:api:chat:evals
• Repo root, chat runtime smoke: bun run test:api:chat:runtime-smoke
• Repo root, targeted API HITL regressions: bun run test:api:chat:hitl
• Repo root, targeted web HITL regressions: bun run test:web:chat:hitl
• Repo root, answer-quality plus persisted experiments review lane: bun run test:api:chat:review
• Repo root, persisted Mastra experiments only: bun run test:api:chat:experiments
• Repo root, runtime answer quality only: bun run test:api:chat:answer-quality
• API app directly, deterministic chat evals: cd apps/api && bun run test:chat:evals
• API app directly, chat runtime smoke: cd apps/api && bun run test:chat:runtime-smoke
• API app directly, targeted HITL regressions: cd apps/api && bun run test:chat:hitl
• API app directly, persisted Mastra experiments: cd apps/api && bun run test:chat:experiments
• API app directly, runtime answer quality: cd apps/api && bun run test:chat:answer-quality

Targeted verification after chat changes

• cd apps/api && bun run test:chat:evals
• cd apps/api && bun run test:chat:runtime-smoke
• cd apps/api && bun run test:chat:hitl
• cd apps/api && bun run check-types
• cd apps/web && bun run test:chat:hitl

Which lane catches what

• Use targeted API/web regression tests for concrete UI, transcript, HITL slideover, action-state, and persistence bugs. Those failures are usually too structural and deterministic to trust to LLM-judge evals.
• Use deterministic chat evals for orchestration-path, focus-mode, response-template, selected-skill, and HITL lifecycle policy regressions.
• Use runtime smoke for real-stack chat execution against bounded fixture/provider boundaries.
• Use Local Connector Prompt Packs when you need gitignored, developer-local prompt packs against the actual connectors available in a local web session.
• Use answer-quality plus persisted experiments as a review lane when prompts, orchestration, scorers, or model-selection behavior changes.
• For merge readiness, do not substitute the review lane for the blocking suite. The canonical gate remains bun run test:local.

Local connector prompt packs

The local connector prompt-pack lane generates natural broad, cross-reference, discovery, fact-seeding, positive, opportunistic, negative, and general-knowledge prompts from the logged-in local connector inventory. It then runs selected cases through real private chats and the production chat route, writing ignored artifacts under tmp/connector-chat-prompts/.

Use this lane to catch provider drift, source-routing mistakes, HITL information-request regressions, and answer-quality issues that deterministic fixtures cannot see. Keep the artifacts local and private; promote only neutralized structural lessons into committed eval fixtures.

Mocking live data correctly

Live-query evals should mock provider-shaped data at the same dependency seam the production canonical tools use. Do not replace private preparation, planner normalization, the private live-evidence loop, or synthesis-preparation contracts with fixture-only orchestration.

For natural-language date prompts, pin currentTurnNow and currentTurnTimeZone on the prompt variant. If the regression is about missing planner bounds, leave the planner fixture step without timeMin/timeMax; the eval should prove production orchestration inferred the window before provider-shaped fixture rows were filtered. Keep decoy rows in the fixture, such as stale same-channel data and current wrong-channel data, so the eval fails when scope or time constraints are dropped.

Catalog Layout

• Scenarios: apps/api/src/mastra/evals/scenarios/
• Fixtures: apps/api/src/mastra/evals/fixtures/
• Raw synthetic fixture assets such as CSV files: apps/api/src/mastra/evals/fixtures/**/data/
• Dataset builders: apps/api/src/mastra/evals/datasets/
• Persisted experiment catalog and runner: apps/api/src/mastra/evals/experiments/
• Deterministic workflow runner: apps/api/src/mastra/evals/runners/chatWorkflowEvalRunner.ts
• Deterministic HITL lifecycle runner: apps/api/src/mastra/evals/runners/chatHitlLifecycleEvalRunner.ts
• Release gates and scorer thresholds: apps/api/src/mastra/scorers/
• Gate-policy env access: apps/api/src/env.ts

Use one scenario file per user-facing case. Add both expected and intentionally wrong or over-broad prompt variants when that improves regression coverage.

Current Focus-Mode Matrix

The deterministic chat eval pack now includes explicit focus coverage for:

• single file, non-tabular
• multiple files, non-tabular
• single file, tabular CSV
• multiple files, tabular CSV
• folders
• library
• connectors

Focused scenarios also include explicit “not present in the focused data” prompts so regressions in no-evidence handling are caught separately from broad off-scope prompts.

Current HITL Coverage

The deterministic chat eval pack now includes explicit information-request lifecycle coverage for:

• initial input-required blocking
• submit and resume with bounded form inputs
• source-choice resume through Mastra workflow suspend/resume
• cancel
• expire
• source-choice missing-input HITL before broad private live retrieval
• source-choice resume with the selected source set applied structurally to orchestration
• source-choice safe fallback through server-owned source values
• bounded detail-clarification HITL for timeframe, subject, and requirements intents
• user-disabled optional clarification behavior, which should continue with server-owned best judgment instead of opening HITL
• source-choice cancel fail-closed behavior
• no-nuisance source-choice coverage for explicit single-source, explicit multi-source, and all-connected-source prompts

Workflow-origin suspend/resume plumbing is covered both as reusable infrastructure and by the live private chat source-choice/detail-clarification flows. Focused connector chats still do not use a choose-sources clarification request; connector focus chosen at chat creation is authoritative.

Manual visual verification for HITL changes should include the live source-choice slideover, a detail-clarification slideover, submit/resume, regenerate-and-resume, Use best judgment, the slideover settings-menu disable action, cancel, refresh-while-pending, desktop/mobile widths, light/dark mode, the Ask clarifying questions setting on and off, Activity input event, no-nuisance prompts that should not open HITL, and an optional-details prompt-injection check such as Talk like a pirate proving details scope retrieval/answer focus without overriding response style or policy.

The executable eval suite also now covers:

• response-template honoring in the deterministic orchestration lane
• selected-skill honoring in the deterministic orchestration lane
• blocking chat runtime smoke answers that execute the real focused tabular/DuckDB path from committed fixture files and the real multi-account Gmail live path from mocked provider boundaries
• canonical live-domain tool routing for connector-agnostic mail and Jira work-item prompts
• natural-language temporal workspace-chat prompts that exercise production Mastra preparation before mocked live-query data is filtered
• persisted Mastra dataset/experiment runs for the deterministic workflow, HITL lifecycle, and chat runtime smoke suites
• Slack now participates in the canonical workspace-chat eval packs, so this lane includes deterministic workflow routing, executable runtime-smoke, and answer-quality coverage for Slack in addition to the targeted connector suites and real-provider Slack smoke path

Private eval runners must apply the same scenario-scoped connected-live tool and provider context before preparation. A workflow eval, answer-quality eval, or local diagnostic that skips that context can accidentally query a connector the scenario intentionally made unavailable. Relative-date eval prompts should also pin currentTurnNow and currentTurnTimeZone in the scenario data so today, tomorrow, and similar prompts do not drift with the machine clock.

Runtime answer-quality review now uses LLM judges for ordinary answer-bearing prompts only:

• answer relevancy
• prompt alignment
• private hallucination against grounded context

Structural no-results, HITL, response-template, and selected-skill assertions now stay in the deterministic workflow, HITL lifecycle, and runtime-smoke lanes instead of being presented as open-ended answer quality.

Live connector coverage must fail closed:

• If any eligible in-scope live connector cannot be checked, the runtime should surface an incomplete-coverage path instead of answering from the surviving subset as though coverage were complete.
• Objective recency requests such as “latest email” should use an explicit planner-selected live query mode, not a targeted search that silently widens into unrelated recent content when there is no match.

Gate Configuration

• CHAT_EVAL_BLOCK_ANSWER_QUALITY=false is the canonical env toggle for whether future runtime answer-quality evals should fail the build.
• Default: false
• Current effect:
  • pull-request and push CI still blocks on deterministic orchestration evals, targeted HITL regression tests, and the chat runtime smoke lane
  • runtime answer-quality evals require provider credentials, so the GitHub job skips pull_request events and runs only from trusted dev pushes, weekday schedules, or reviewed manual dispatches
  • runtime answer-quality evals remain a review lane by default and only fail when this flag is true
• Set it in:
  • apps/api/.env for local operator runs
  • .github/workflows/chat-orchestration-evals.yml for GitHub CI
• Related eval env vars:
  • CHAT_EVAL_BEDROCK_MODEL_ID
  • CHAT_EVAL_PRIVATE_SYNTH_MODEL_IDS
  • CHAT_EVAL_PUBLIC_MODEL_ID
  • CHAT_EVAL_PUBLIC_MODEL_IDS
  • CHAT_EVAL_PUBLIC_PROVIDER_API_BASE_URL
  • CHAT_EVAL_PUBLIC_PROVIDER_API_KEY
  • CHAT_EVAL_VERBOSE_OUTPUT
  • CHAT_RUNTIME_EVAL_SAMPLE_RATE

Persisted Experiments

• The canonical persisted experiment runner uses Mastra datasets and dataset.startExperiment(...), not an app-owned experiment format.
• Deterministic persisted experiments currently cover:
  • chat-workflow-evals
  • chat-hitl-lifecycle-evals
  • chat-runtime-smoke-evals
• Those runs persist dataset versions, experiment ids, and item-level scores to the configured Mastra datasets storage so operators can compare runs over time.
• The persisted experiment lane uses the same scenario catalog and deterministic release gate as the blocking runEvals(...) lane; it adds traceable dataset/experiment history rather than a separate policy model.
• Mastra Studio datasets and experiments stay empty until you actually run the persisted experiment lane (bun run test:api:chat:experiments or cd apps/api && bun run test:chat:experiments). Ordinary app startup and deterministic eval runs do not backfill that storage automatically.

Mastra Studio Notes

• Mastra Studio only shows tools that are registered on the top-level Mastra instance. In this repo, the canonical live tools are now registered in apps/api/src/mastra/index.ts, so Studio should show mail-live-query, contact-live-query, calendar-live-query, workspace-chat-live-query, and work-item-live-query instead of only research-query-tool.
• If Studio does not show new tools, first confirm the tool is registered in the top-level tools map, not only called indirectly from a workflow or helper module.
• Studio may show some registered scorers as not attached to any agent or workflow step. That is expected for CI, experiment, and trace-scoring scorers that are registered globally for manual runs and persisted experiments.
• Selected private preparation, evidence preparation, and live-evidence workflow steps now attach live scorers through createStep({ scorers }). Keep any additional workflow scorer expansion anchored in docs/features/chat/spec.md so Studio-visible scoring stays intentional.

Model Comparison And Answer Inspection

For the high-level model role map, current defaults, and promotion path, start with Chat Models And Evals. This page keeps the eval runbook focused on commands and triage.

• Runtime answer-quality evals can compare model ids without changing the product default:
  • CHAT_EVAL_PRIVATE_SYNTH_MODEL_IDS=model-a,model-b
  • CHAT_EVAL_PUBLIC_MODEL_IDS=model-a,model-b
• Private answer-quality comparison automatically includes the current API-owned private synthesizer model from validated runtime config. Explicit private model ids are treated as additional comparison candidates, not replacements for that current-runtime baseline.
• When those vars are set, the runner executes the same scenario pack once per listed model id and prints separate average scores for each profile plus a comparison summary at the end.
• The comparison summary includes target-avg and target-p95 from Mastra item execution timestamps, plus eval-wall for total local run time including scorers. Use target timings for user-response tradeoffs and wall time for local/CI cost planning.
• Leave those vars empty for the normal single-profile run.
• To inspect the actual rendered answers while iterating, set CHAT_EVAL_VERBOSE_OUTPUT=true.
• Use comparison runs to validate API-owned private runtime config choices and CMS-owned public model catalog choices before changing the canonical configuration.
• Eval-only comparison ids do not change production behavior by themselves:
  • to promote a private candidate model into runtime, add an eval-approved primary or fallback chain entry in leverage-runtime.config.*
  • to promote a public candidate model into runtime, add or enable it in the seeded CMS public model catalog while keeping provider runtime wiring in API runtime config

How To Review A Failure

1. Start in Mastra Studio and inspect the failing scorer trace.
2. Classify the regression as model, prompt, grounding, orchestration, or data related.
3. For grounded private-answer prompt-alignment failures, confirm the scorer input includes the same private grounding context used by hallucination scoring before treating the answer as fabricated.
4. If the failure is actually UI/state/persistence/HITL behavior, move to the targeted API/web regression suites instead of trying to force-fit it into the LLM-judge lane.
5. Fix the underlying policy, prompt, fixture, or scoring expectation rather than adding route-local heuristics.
6. Re-run the affected scenario set and confirm the release gate still passes.
7. Update docs/features/chat/spec.md or the matching story when the canonical behavior changes.
8. If the feature spec/story/tests and this operator page disagree or leave the intended behavior ambiguous, stop, clarify the direction, and update the canonical docs plus this mirror page in the same branch.

Release Gate Summary

• chat-orchestration-ci is the deterministic gate for the scenario-driven chat workflow eval pack and is the current blocking gate for GitHub CI in this slice.
• private-orchestration-ci remains available for the older deterministic scorer baseline.
• private-answer-quality-pre-release and public-answer-quality-pre-release now run through the manual answer-quality lane in GitHub.
• CHAT_EVAL_BLOCK_ANSWER_QUALITY is the canonical release-policy toggle for promoting that manual answer-quality lane from report-only to blocking.
• Answer-quality remains non-blocking by default. Turn the env toggle on only after you are ready for the answer-quality workflow to fail-close on the documented thresholds.
• The private live lane can still fail intermittently on the Bedrock judge-side scorer workflow with upstream 500s. Treat that as scorer-provider stability debt unless a deterministic or rendered-answer regression accompanies it.
• Threshold values and fail-close behavior are defined in docs/features/chat/spec.md.
• Segment-level or section-level grounding is available for mixed evidence/general-knowledge answers through privacy-safe groundingSegments. Eval review should still treat missing, misleading, or unclear source-backed versus general answer portions as a regression risk.

Current Limits And Follow-Ups

• Runtime answer-quality scorer retry handling is hardened at the Mastra runner boundary, but judge/scorer calibration is still follow-up work. Keep that work grounded in broader real-world review sets and avoid tuning prompts or thresholds against the current fixture pack just to lift scores.