Chat Models And Evals
Chat orchestration uses model roles, not one global model. This page is the short operator/developer map; docs/features/chat/spec.md remains the canonical source for requirements and release policy.
Current Model Policy
Section titled “Current Model Policy”- Private chat LLM calls must stay on Bedrock. There is no public-provider fallback for private prompts, retrieved connector data, tabular results, embeddings, or private eval judging.
- Public chat uses the CMS Chat Behavior
Enable Public Chatswitch, CMS approved-model policy, API runtime provider wiring, and a valid public-provider credential slot. Public models are selectable only in public mode, are explicitly chosen from the policy-resolved catalog, and must never receive private connector evidence. - Private role model chains are loaded from validated API runtime config. Customer CMS does not expose private role choices, private model presets, fallback chains, thinking budgets, provider reasoning summaries, chain-of-thought controls, credentials, or arbitrary private model ids.
- API-owned validation rejects any private primary/fallback model entry that is not eval-approved for that role. Empty fallback arrays mean no fallback; if the primary cannot be used and no approved fallback exists, private chat fails closed.
- API code owns reasoning-effort policy and private tool availability. Customer CMS does not expose private tool enable/disable controls.
Private Role Defaults
Section titled “Private Role Defaults”| Role | Current default runtime intent | Current default model id |
|---|---|---|
| Intent classifier | Fast structured private-evidence classification | us.anthropic.claude-haiku-4-5-20251001-v1:0 |
| Guardrail classifier | Fast structured prompt-injection and policy classification | us.anthropic.claude-haiku-4-5-20251001-v1:0 |
| Planner | Higher-reliability private retrieval/tool planning | us.anthropic.claude-sonnet-4-6 |
| Tabular planner | Higher-reliability SQL/tabular planning | us.anthropic.claude-sonnet-4-6 |
| Synthesizer | Durable-stream-safe grounded private answer synthesis | us.anthropic.claude-haiku-4-5-20251001-v1:0 |
| Eval judge | Strong private answer-quality judging | us.anthropic.claude-sonnet-4-6 |
The private synthesizer default is Claude Haiku 4.5 while chat stream replay runs through Mastra DurableAgent beta. The previous Mistral Large 3 default remains an explicit comparison candidate, but it must be revalidated on the durable stream path before being promoted again.
Public Model Defaults
Section titled “Public Model Defaults”The seed-managed public catalog currently starts with OpenAI-compatible public chat models:
| Rank | Model id | Default |
|---|---|---|
| 1 | gpt-5.5 |
Yes |
| 2 | gpt-5.4 |
No |
| 3 | gpt-5.2 |
No |
Public catalog identity/runtime wiring is seed/API-runtime-owned. Customer CMS policy can enable seeded providers after credential validation and can enable, order, and choose defaults for approved model rows. Public model choices do not affect private orchestration.
For local development, bun run init runs the customer CMS seed path after local Mongo is available.
If the Public Chat Providers or Public Chat Models collections are empty on a fresh machine, start
local Docker services and rerun bun run init.
Authenticated users may also save a default public provider/model preference. That preference lives in user settings, applies only to future public chat drafts, and never overrides the runtime catalog or send-time validation. If the saved model is no longer available, the web app falls back to the catalog default or first enabled model without clearing the saved preference.
Public model auto-selection is not implemented yet. Current public sends use explicit allowed provider/model choices that the API revalidates at execution time.
If Chat Behavior disables public chat, or if no enabled credential-backed public model resolves, the web runtime config is private-only and the API rejects public create/send attempts.
Embeddings
Section titled “Embeddings”Connector/file embeddings use Bedrock amazon.titan-embed-text-v2:0 by default. That path is separate from chat synthesis roles and is used for indexed connector retrieval and vector-query embeddings. Evaluate a different Bedrock embedding model only when a retrieval-quality requirement, language requirement, or modality requirement justifies the change.
How Resolution Works
Section titled “How Resolution Works”apps/apidefines approved private model entries and default role chains.- The API loads non-secret runtime config with c12 and validates it strictly with Zod.
- Runtime config may assign only approved primary/fallback entries for each private role.
- Per-turn request context carries typed resolved role chains to Mastra agents.
- Private agents resolve Bedrock models through the shared private Bedrock resolver and fail closed when no approved model chain can be used.
This shape keeps Studio-visible Mastra agents/workflows while keeping private model governance API-owned and Bedrock-only.
Eval Lanes
Section titled “Eval Lanes”| Lane | Command | Purpose |
|---|---|---|
| Deterministic workflow evals | bun run test:api:chat:evals |
Blocks wrong routing, wrong tool/path selection, HITL lifecycle regressions, response-template regressions, selected-skill regressions, focus-mode policy drift, and natural-language live-query temporal routing. |
| Runtime smoke | bun run test:api:chat:runtime-smoke |
Exercises bounded real runtime paths such as focused CSV/DuckDB and mocked live-provider boundaries. |
| HITL regressions | bun run test:api:chat:hitl and bun run test:web:chat:hitl |
Verifies API/web information-request submit, resume, cancel, expire, request-panel, and transport behavior. |
| Answer-quality review | bun run test:api:chat:answer-quality |
Uses LLM judges for answer-bearing prompts and prints score/timing summaries. Non-blocking by default unless CHAT_EVAL_BLOCK_ANSWER_QUALITY=true. |
| Persisted experiments | bun run test:api:chat:experiments |
Persists Mastra dataset/experiment history for Studio review and run comparison. |
| Local merge gate | bun run test:local |
Runs the broad local gate, including chat eval/runtime/HITL lanes. |
Use deterministic lanes for structural behavior. Use answer-quality and persisted experiments for prompt/model/scorer review. Do not tune prompts or thresholds only to satisfy the current fixture pack.
For live-query temporal prompts, deterministic evals pin the prompt turn clock and mock provider-shaped data at the canonical tool dependency seam. Planner fixture steps should omit timeMin/timeMax when the behavior under test is shared temporal inference rather than fixture filtering.
Model Comparison Workflow
Section titled “Model Comparison Workflow”- Private synthesizer comparison uses
CHAT_EVAL_PRIVATE_SYNTH_MODEL_IDS=model-a,model-b. - Public comparison uses
CHAT_EVAL_PUBLIC_MODEL_IDS=model-a,model-b. - The private comparison runner includes the current API runtime synthesizer baseline when runtime config is valid.
- The comparison summary includes target average and p95 timings plus total eval wall time.
- Promote a private candidate only by adding it to the API-owned approved role definitions and runtime config after eval approval; do not change product defaults from an eval-only env var.
- Promote a public candidate through the seeded public catalog plus CMS enable/default/order policy.
Current Watchlist
Section titled “Current Watchlist”us.anthropic.claude-haiku-4-5-20251001-v1:0is the supported private synthesizer default in this repo while DurableAgent stream replay is beta.mistral.mistral-large-3-675b-instructremains an experimental durable-stream recheck candidate because it was the previous non-durable answer-quality winner.openai.gpt-oss-120b-1:0remains an experimental private synthesizer baseline for cost/quality comparison.- Claude Haiku/Sonnet profiles remain role-appropriate alternatives for lower-latency classification or higher-reliability planning/judging.
- Mistral Medium 3.5 is a watchlist item only. Do not seed a guessed Bedrock model id; wait until AWS Bedrock lists a concrete supported id through docs or model-list APIs, or until the team accepts a separate private self-hosted path.
Canonical Sources
Section titled “Canonical Sources”docs/features/chat/spec.mddocs/features/user-settings/spec.mddocs/features/chat/stories/chat-orchestration-implementation.stories.mdapps/api/src/runtimeConfig/apiRuntimeConfig.tsapps/api/src/runtimeConfig/privateChatActiveModelProfiles.tsapps/api/src/runtimeConfig/privateChatEvalModelCandidates.tsapps/api/src/runtimeConfig/privateChatRuntimeModelMetadata.tsapps/api/src/runtimeConfig/privateChatRuntimeRegistry.tsapps/api/src/chats/chatOrchestrationConfig.tsapps/api/src/chats/privateRoleModelConfig.tsapps/api/src/mastra/agents/privateBedrockModelResolver.tsapps/api/src/mastra/evalsapps/api/src/mastra/scorers