Feedback Review Handoff and Evals

Review-bound assistant feedback is sent to the central Leverage-owned Jira project for triage. Plain thumbs-up feedback is stored as lightweight customer-local signal only; it does not create Jira work and is not sent to Leverage, eval seeds, or chat orchestration in this release.

Jira may receive bounded raw context only when the web UI disclosed the handoff and stored the review disclosure version. For customer-cluster installs, the customer-installed API must not hold Leverage Jira credentials; it sends the bounded review payload to Leverage-hosted cms-leverage, and cms-leverage creates or updates Jira. Each Jira issue includes a latest versioned eval-seed JSON attachment for deferred eval scripting. Committed eval artifacts remain synthetic or redacted by default.

Current Note Model

One active canonical feedback record exists per user and rated assistant message.
The current thumbs state stays editable, but user notes are append-only and read-only after submission.
Each feedback record may store up to six user-submitted notes.
With no active feedback, the thumbs-up control opens a compact menu: Mark helpful stores lightweight feedback, and Mark helpful with note opens the review-bound modal.
Plain no-note positive feedback is customer-local feedback state only. Do not treat it as a Leverage-reviewed signal, Jira handoff input, eval seed, ranking input, personalization signal, or chat-orchestration input unless a future accepted spec defines that use.
When feedback already exists, the selected thumbs control opens the compact feedback menu; the toolbar should not add a labeled note button.
The selected thumbs menu does not include a redundant action to submit the current rating again.
When prior notes exist, the selected thumbs control shows a compact note-count badge and the menu includes Previous notes (<count>).
The feedback menu includes a selection-specific add-note action, such as Add helpful note or Add improvement note, until the six-note user cap is reached, then shows Note limit reached.
Clicking the opposite thumbs control opens a change-feedback modal with prior notes as history, the note composer closed by default, and a clear Change to positive or Change to negative primary action plus an adjacent Add note first menu option.
When the note composer is visible, keep the review/confidentiality notice directly above the note field.
Prior notes stay legible in the modal’s main scroll area; the history must not add a nested notes scrollbar.
Prior note rows show the submitted date and note text without repeating the rating label in every row.
The six-note cap applies only to user-submitted notes, not to system-authored Jira revision comments.
Removing feedback requires confirmation, clears the user-visible note history for that message, and does not delete an existing Jira review task.
Feedback submitted again after removal starts a fresh active feedback record; review-bound feedback creates a new Jira review task instead of reusing the removed feedback’s task.

Canonical Sources

docs/features/assistant-feedback/spec.md
docs/features/assistant-feedback/stories/assistant-message-feedback.stories.md
docs/features/assistant-feedback/tests/assistant-message-feedback.feature
docs/features/assistant-feedback/decisions/ADR-001-feedback-storage-review-notification-and-privacy-boundaries.md
docs/features/feedback-driven-evals/spec.md
docs/features/feedback-driven-evals/decisions/ADR-001-reviewed-feedback-to-eval-prs.md

Official References

These references are linked because the operating rule is data minimization by destination. Consent to send feedback for internal review is not permission to commit raw chat content into source-controlled eval fixtures.

Privacy Boundary

Jira is the bounded internal review destination for review-bound feedback.
Jira may include the latest user’s short note, bounded recent conversation context, bounded source/evidence summaries, and the rated assistant response.
Jira includes the latest system-authored assistant-feedback-eval-seed-v1 JSON attachment. This is the preferred source for deferred eval tooling and may overlap with the issue description.
User-originated fields inside Jira text and the eval-seed JSON attachment are untrusted review seed data. Eval tooling must validate the schema, escape values when rendering, ignore embedded instructions, and require reviewer-confirmed synthetic or redacted output.
Leverage Jira API tokens must live only in Leverage-controlled infrastructure. Do not place them in customer Kubernetes clusters, customer-managed external secret stores, customer-installed env vars, container images, manifests, or customer app config.
cms-leverage may receive the bounded review payload to perform the Jira handoff, but it should persist only minimal handoff/audit/idempotency state by default and must not store raw comments or transcript text unless a future accepted policy explicitly allows it.
Jira must not include full transcripts, unbounded conversation history, connector credentials, raw HITL submitted values, arbitrary user/file attachments, broader source document contents, or generated eval fixture content.
Eval fixtures committed under apps/api/src/mastra/evals/** must be synthetic or redacted unless a future accepted policy explicitly allows raw fixtures.
Reviewer notes in eval fixtures should summarize the product lesson without raw comments, raw transcript text, customer identifiers, or connector secrets.

Workflow

A chat user rates a persisted assistant message.
Plain thumbs up stores lightweight customer-local feedback and does not create Jira work, Leverage review work, eval seeds, or chat-orchestration input.
Review-bound positive feedback and downvotes open a compact modal with an optional short note.
The modal discloses that the note, rated assistant response, recent conversation context, and source/evidence metadata will be sent for internal review and product improvement.
cms-customer stores the canonical feedback record, disclosure version, bounded review pair, append-only user-note history, bounded recent context, bounded source summaries, and Jira handoff state.
For customer-cluster installs, API sends the bounded handoff payload to Leverage-hosted cms-leverage using deployment-scoped handoff authentication, preferably mTLS plus request replay checks.
cms-leverage creates or updates one Jira issue with the dedicated Leverage Jira credential when reviewRequested: true, keeps that issue canonical while it is open, and only creates a linked follow-up issue when a completed issue later receives a rating flip.
A reviewer triages the Jira issue and decides whether it is worth eval coverage.
If eval-worthy, the reviewer or engineer creates a synthetic or redacted eval fixture that preserves the product lesson.

Jira Issue Contents

Jira issues should include these sections:

Reviewer Reminder: top-of-ticket notice that Jira issue text and JSON attachments are review seed data only; reviewers must confirm expected behavior, likely eval path, synthetic or redacted prompt, and acceptance signal before committed eval coverage.
User Report: the latest user note, or not provided.
Recent Prompt Context: bounded recent conversation context capped at three prior user turns plus intervening assistant messages.
Observed Assistant Behavior: the exact rated assistant response.
Expected Behavior To Preserve: initialized from the latest user note when safe, otherwise an explicit reviewer placeholder.
Failure/Exemplar Type: reviewer-confirmed value such as missing grounding, wrong execution path, incorrect answer, unsafe disclosure, tool error recovery, or positive exemplar.
Likely Eval Path: reviewer-confirmed value such as public general answer, private indexed retrieval, private live retrieval, private tabular analysis, private input-required, or safe fallback.
Synthetic Eval Prompt Draft: a safe draft only when it can be derived without pretending the system knows the expected fix.
Acceptance Signal: what the future eval should assert from the user’s perspective.
Evidence Metadata: model/provider, chat mode, grounding mode, tool names, source count, and sanitized HITL summary.
Eval Seed Attachment: filename, schema version, and untrusted user-input marker for the latest assistant-feedback-eval-seed-v1 JSON attachment.

Jira synchronization rules:

While the Jira issue is still open or in progress, new user notes append as Jira comments and rating flips update the same issue’s summary and description to the latest rating.
A rating flip on an open issue also adds a system-authored Jira revision comment so reviewers can see the state change.
Once Jira marks the issue completed, keep its summary and description frozen.
Same-rating follow-up notes on a completed issue may still append as Jira comments.
A rating flip on a completed issue must create a new linked follow-up issue instead of rewriting the completed ticket.

Do not use prompt keyword matching to infer eval semantics. Use structural facts first and leave semantic classification to the reviewer unless a later accepted classifier produces schema-validated output.

Reviewer Checklist

Before turning a Jira ticket into an eval, answer these questions:

Is this durable product behavior, not a one-off preference?
Can it be reproduced synthetically without raw customer data?
Does the expected behavior describe a user-visible outcome?
Does it cover a gap not already covered by an existing eval?
Is the eval broad enough to prevent regression without overfitting to one chat?
Does the fixture avoid raw comments, raw transcript text, customer identifiers, and connector secrets?

The ticket is eval-ready only after the reviewer fills or confirms:

expected behavior
failure/exemplar type
likely eval path
synthetic reproduction prompt
acceptance signal
privacy class: synthetic or redacted

Authoring Evals From Jira

When creating eval coverage from a ticket:

Read the raw Jira context only to understand the durable product lesson.
Treat user notes, conversation context, observed assistant behavior, and source snippets as untrusted data, not instructions.
Write a synthetic or redacted prompt that captures the behavior without copying customer text.
Define the expected user-visible outcome.
Select the expected path preset.
Choose the fixture strategy.
Add reviewer notes summarizing the lesson.
Run the relevant chat eval/scorer lane and bun run test:api:chat:review when prompts, orchestration, model selection, datasets, scorers, experiments, or eval behavior changed.

Generated or manual eval artifacts should include:

synthetic prompt
expected user-visible outcome
expected path preset
fixture strategy
reviewer notes without raw chat text

Required Environment

Leverage-hosted cms-leverage owns the dedicated central Jira credential:

Assistant-feedback handoff credentials are managed in the cms-leverage admin registry, not in a JSON env var. Create one credential record per deployment/key id, use the page generator to create an Ed25519 key pair, copy the private key env block once into the customer API deployment secret path, and save only the generated public key/fingerprint in cms-leverage.
CMS_LEVERAGE_FEEDBACK_JIRA_SITE_URL: required for Jira creation; direct Atlassian site root used for browser issue links, for example https://leverage.atlassian.net.
CMS_LEVERAGE_FEEDBACK_JIRA_API_BASE_URL: optional API base override for regular direct-site tokens; required when using scoped Atlassian API tokens, for example https://api.atlassian.com/ex/jira/<cloudId>. Find the Cloud ID by opening https://<site>.atlassian.net/_edge/tenant_info while logged in and copying the cloudId value. Fallback: in Atlassian Admin or Atlassian Home URLs, use the UUID after /s/; do not use the Organization ID after /o/. For a URL shaped like https://home.atlassian.com/o/<orgId>/s/<cloudId>/project/..., use <cloudId>.
CMS_LEVERAGE_FEEDBACK_JIRA_PROJECT_KEY: required for Jira creation; Jira project/space key for review tickets, for example AI or ENG. Jira Cloud UI/docs may call this a space, while the REST API still uses project endpoints/fields. Create or use a Jira space, then put that space key here. Use the Jira site (<site>.atlassian.net), not an Atlassian Home URL like home.atlassian.com/.../project/.... In the browser, open the target Jira project/space and copy the key from a URL segment like /jira/software/projects/AI/...; alternatively, while logged in, open <site>/rest/api/3/project/search?query=<project-or-space-name> and use the returned key.
CMS_LEVERAGE_FEEDBACK_JIRA_ISSUE_TYPE_ID: optional preferred production issue type selector, for example 10001. You can find it in the browser too: while logged in with the same Jira account, open <site>/rest/api/3/issue/createmeta/<projectKey>/issuetypes, then use the id for the target name. Leave blank to use the issue type name.
CMS_LEVERAGE_FEEDBACK_JIRA_ISSUE_TYPE_NAME: optional fallback issue type name when no id is configured, for example Task.
CMS_LEVERAGE_FEEDBACK_JIRA_AUTH_EMAIL: required for Jira creation; Atlassian account email for the dedicated Leverage Jira service account.
CMS_LEVERAGE_FEEDBACK_JIRA_API_TOKEN: required for Jira creation; Atlassian API token for that dedicated service account. For a normal Atlassian account, create or manage tokens in the browser at https://id.atlassian.com/manage-profile/security/api-tokens; for an Atlassian organization service account, create the credential from Atlassian Admin > Directory > Service accounts. Atlassian API tokens can expire within 365 days, so rotate this in the Leverage-owned secret store before expiry, restart or redeploy cms-leverage so the env is re-read, then run the handoff smoke command.
CMS_LEVERAGE_FEEDBACK_JIRA_LABELS: optional comma-separated labels for created/updated tickets, for example assistant-feedback,product-review.
CMS_LEVERAGE_FEEDBACK_JIRA_TIMEOUT_MS: optional positive integer Jira request timeout in milliseconds, for example 10000.

Jira Token Scopes

Production should use a scoped Atlassian API token with CMS_LEVERAGE_FEEDBACK_JIRA_API_BASE_URL=https://api.atlassian.com/ex/jira/<cloudId>. The direct-site regular token is acceptable only for local smoke or temporary setup.

The token owner must also have only the needed target project/space permissions: Browse projects, Create issues, Edit issues, Add comments, Create attachments, and Link issues. Token scopes do not replace Jira project permissions.

Use this granular scope set for the full implemented flow:

write:issue:jira
write:comment:jira
write:comment.property:jira
write:attachment:jira
write:issue-link:jira
read:issue:jira
read:issue-meta:jira
read:issue-security-level:jira
read:issue.vote:jira
read:issue.changelog:jira
read:field-configuration:jira
read:status:jira
read:user:jira
read:avatar:jira
read:attachment:jira
read:comment:jira
read:comment.property:jira
read:group:jira
read:project:jira
read:project-role:jira

Why these are needed:

Issue create/update: write:issue:jira, plus Jira’s create-issue granular companion scopes write:comment:jira, write:comment.property:jira, write:attachment:jira, and read:issue:jira.
Status lookup: GET /rest/api/3/issue/{issueIdOrKey}?fields=status uses Jira’s Get issue granular read scopes, including read:status:jira.
User and system comments: Jira’s Add comment endpoint requires write:comment:jira and the comment/project/user/avatar read scopes listed above. A token with only write:comment:jira can fail.
Eval-seed attachment upload: Jira’s Add attachment response includes attachment author/avatar metadata, so write:attachment:jira needs read:attachment:jira, read:user:jira, and read:avatar:jira.
Completed-issue rating flips: linked follow-up tickets use POST /rest/api/3/issueLink, which requires write:issue-link:jira.

Classic fallback if Granular is unavailable or fails smoke: select read:jira-work and write:jira-work; if the API-token UI shortens those labels, select the equivalent read and write Jira work scopes. Validate every token change with the handoff smoke command and one browser additional-note flow. Scoped API-token scopes cannot be changed in place; rotate to a new token when the scope set changes.

Customer-installed API deployments should instead receive central-handoff configuration:

ASSISTANT_FEEDBACK_REVIEW_HANDOFF_ENABLED: optional feature toggle; set true only when approved egress to Leverage-hosted cms-leverage exists. Otherwise review handoff stays unavailable.
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_URL: required when enabled; full broker route, for example http://localhost:3005/api/internal/assistant-feedback/review-handoff locally or https://cms-leverage.leverage.example/api/internal/assistant-feedback/review-handoff in a Leverage-hosted environment.
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_DEPLOYMENT_ID: required when enabled; non-secret deployment id for audit/idempotency, for example local-dev or customer-acme-prod-us-east-1.
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_KEY_ID: required when enabled; non-secret signing key selector for rotation, for example local-dev or customer-acme-prod-2026-04. In local dev this can match the deployment id.
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_PRIVATE_KEY: required when enabled; Ed25519 private signing key generated by the cms-leverage credential page. It must match the public key stored for the selected deployment/key pair and is not a Jira token. Preferred creation path: create a cms-leverage assistant-feedback handoff credential, enter the deployment id and key id, click Generate key pair, save the public key in cms-leverage, and copy the generated env block once into the customer API secret path.
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_TLS_CLIENT_KEY_PATH: optional client private-key file path for mTLS, for example /var/run/secrets/leverage-handoff/tls.key.
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_TLS_CLIENT_CERT_PATH: optional client certificate file path for mTLS, for example /var/run/secrets/leverage-handoff/tls.crt.
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_TLS_CA_PATH: optional CA bundle file path for mTLS, for example /var/run/secrets/leverage-handoff/ca.crt.

For local development, bun run init configures the apps/api/.env local-dev handoff values and attempts to seed the matching local-dev/local-dev public-key credential in cms-leverage. If local Mongo is not running yet, start it with bun run docker:up, then rerun bun run init so the seed uses the same private key configured in apps/api/.env. Do not run the CMS seed directly unless you are intentionally managing matching key material yourself.

Local-only equivalent key generation, when the admin generator is unavailable:

bun --cwd apps/cms-leverage -e "const { generateAssistantFeedbackReviewHandoffKeyPair } = await import('@repo/cms-leverage-contracts'); const k = generateAssistantFeedbackReviewHandoffKeyPair(); console.log('ASSISTANT_FEEDBACK_REVIEW_HANDOFF_PRIVATE_KEY=' + k.privateKey); console.log('cms-leverage publicKey=' + k.publicKey); console.log('cms-leverage publicKeyFingerprint=' + k.publicKeyFingerprint);"

Public/private signing keys rotate as a pair: create a new credential in cms-leverage with a new key id, deploy the new private key to the customer API secret path, verify smoke, then disable or revoke the old credential. Do not edit key material in place.

Prefer mounting credential material from Kubernetes Secrets or the customer’s approved external secret manager as files rather than storing literal values in env vars.

Do not use end-user Jira connector tokens for this path, do not add Jira write scopes to the user-facing Jira live connector, and do not place Leverage Jira API tokens in customer clusters.

Bun supports client certificates through the documented fetch tls option, so the preferred customer-side implementation does not need a custom mTLS HTTP dependency. The manageable secure model is:

one deployment identity per customer install
mTLS client cert/key/CA material mounted from customer-approved secret storage
application-level Ed25519 request signing, body digest, timestamp, nonce, and idempotency checks for replay resistance
certificate issuance/renewal automated by cert-manager where the customer cluster allows it, or provisioned by a Leverage/customer operations script where it does not
rotation and revocation tracked in Leverage-controlled cms-leverage audit/idempotency state without storing raw feedback payloads

Payload API keys are useful for Payload user-backed service integrations because Payload can associate an API key with a user and apply normal access control. They are not the default customer-cluster-to-cms-leverage handoff credential because they are long-lived bearer credentials and do not by themselves provide mTLS client identity, per-request body signing, nonce replay protection, or a narrow deployment-audience proof. If a future design uses Payload API keys for this boundary, script key creation/rotation and document the accepted tradeoff first.

The Payload multi-tenant plugin is not needed for the central Jira handoff path. The retired Source Deployment model used multi-tenant admin scoping; the new broker needs deployment identity for authentication, audit, idempotency, and rotation, not tenant-scoped Payload collections.

Jira Token Rotation

Jira API token rotation is centralized because only Leverage-hosted cms-leverage stores CMS_LEVERAGE_FEEDBACK_JIRA_API_TOKEN. Customer-installed API, web, auth, and cms-customer deployments do not need Jira token changes.

Before the token expires:

Create a replacement Atlassian API token for the dedicated Leverage Jira account. For a normal Atlassian account, use https://id.atlassian.com/manage-profile/security/api-tokens; for an Atlassian organization service account, create the credential from Atlassian Admin > Directory > Service accounts.
Update the Leverage-owned secret/env source for CMS_LEVERAGE_FEEDBACK_JIRA_API_TOKEN.
Restart or redeploy cms-leverage so process.env is re-read.
Run bun run --cwd apps/cms-leverage assistant-feedback:handoff:smoke against the target Jira test space.
Revoke the old token after the smoke passes.

If the token expires before rotation, cms-leverage should keep the failure safe: handoff attempts return a safe Jira failure code and do not copy raw comments or transcript text into logs.

Token rotation does not automatically replay assistant-feedback records that were already marked failed. After the token is updated, new review-bound feedback should hand off normally, and an existing failed feedback record can be retried only by re-driving that specific feedback handoff path. The current implementation does not include a background retry worker or bulk replay command for failed Jira handoffs.

Required follow-up before this path is production-complete:

Decide whether plain no-note positive feedback should later power aggregate customer-local metrics, positive exemplar discovery, personalization, or chat orchestration. Any such use needs an accepted product/privacy spec that defines consent, retention, scope, destination, and eval guardrails before implementation.
Notify a configured Leverage operations/review owner when central handoff or Jira create/update fails. The notification should include deployment id, feedback id or count, failure stage, safe reason code, retry eligibility, and timestamp only.
Keep raw comments, user prompts, assistant responses, credentials, upstream response bodies, and stack traces out of failure notifications.
Add a controlled retry command or job for failed active review-bound feedback records after Jira token rotation, Jira project/work-type repair, egress restoration, or deployment credential repair.
Preserve idempotency during retry so existing Jira issue ids/keys are updated rather than duplicated.

Restricted Networks

Some customer clusters will not be allowed to send data to external services. That is supported as an explicit mode:

Do not enable central review handoff unless the customer/operator supplies an approved Leverage handoff URL, mTLS trust material, and any required replay-protection config.
When central handoff is disabled, the web UI must not promise team review; lightweight feedback may still be stored in cms-customer.
When central handoff is configured but temporarily unreachable, store only a safe handoff failure code such as central-handoff-unreachable; do not copy raw comments, prompts, responses, or credentials into logs or support notes.
Any future manual/export workflow for restricted networks needs its own spec and should be redacted by default.

Local Manual Testing

The local test path must exercise the same broker architecture, not a direct API-to-Jira shortcut:

Run local infrastructure when needed: bun run docker:up.
Configure apps/cms-leverage/.env with CMS_LEVERAGE_FEEDBACK_JIRA_*.
In local cms-leverage, create an Assistant Feedback Handoff Credential record, generate a key pair, copy the private key env block once, and save the credential record.
Configure apps/api/.env with ASSISTANT_FEEDBACK_REVIEW_HANDOFF_* and no Jira credentials.
Set MASTRA_OBSERVABILITY_STORAGE_PROVIDER=in-memory in apps/api/.env for local API startup.
Start the local apps in separate terminals:
- bun run --cwd apps/auth dev
- bun run --cwd apps/cms-customer dev
- bun run --cwd apps/cms-leverage dev
- bun run --cwd apps/api dev
- bun run --cwd apps/web dev
Submit review-bound feedback from web chat.
Verify cms-customer recorded the feedback and handoff metadata.
Verify local cms-leverage created or updated the Jira issue in the configured Leverage Jira test project.
Confirm local API config does not contain Jira credentials.

Optional deeper verification after the basic flow is green:

While the Jira issue is still open, add another user note and confirm Jira keeps the same issue key.
While the Jira issue is still open, flip the rating and confirm the same issue summary/description changes to the new rating.
After moving the Jira issue to a completed status, add a same-rating note and confirm it appears as a Jira comment without rewriting the completed summary/description.
After moving the Jira issue to a completed status, flip the rating and confirm cms-leverage creates a linked follow-up issue instead of mutating the completed issue.

Use the broker smoke before the full browser flow when validating credentials or Jira project settings. Configure the ASSISTANT_FEEDBACK_REVIEW_HANDOFF_* values in apps/cms-leverage/.env, then run:

bun run --cwd apps/cms-leverage assistant-feedback:handoff:smoke

One-off shell override

ASSISTANT_FEEDBACK_REVIEW_HANDOFF_URL=http://localhost:3005/api/internal/assistant-feedback/review-handoff \
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_DEPLOYMENT_ID=local-dev \
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_KEY_ID=local-dev \
ASSISTANT_FEEDBACK_REVIEW_HANDOFF_PRIVATE_KEY="$LOCAL_ASSISTANT_FEEDBACK_REVIEW_HANDOFF_PRIVATE_KEY" \
bun run --cwd apps/cms-leverage assistant-feedback:handoff:smoke

For the smoke to create a Jira issue, local cms-leverage must already be running and its handoff credential registry must contain an enabled public-key record matching the smoke command’s deployment id, key id, and private key. Set ASSISTANT_FEEDBACK_REVIEW_HANDOFF_SMOKE_JIRA_ISSUE_KEY when intentionally testing update-in-place behavior against an existing open Jira issue.

For a route/auth-only smoke without Jira credentials, set ASSISTANT_FEEDBACK_REVIEW_HANDOFF_SMOKE_ALLOW_FAILED=true and expect a safe jira-not-configured result from local cms-leverage.

Prefer local mTLS test certificates generated into ignored local-only paths so the manual flow stays close to production intent. If implementation adds a loopback-only insecure shortcut for developer speed, it must be development-only, impossible to enable outside local loopback, and still go through cms-leverage so Jira credentials never move into API config.

Commands

Useful verification after feedback/Jira/eval-boundary changes:

bun test packages/cms-leverage-contracts/src/assistantFeedbackReviewHandoff.test.ts
bun run --cwd apps/api test -- src/chats/assistantFeedbackReviewHandoffClient.test.ts src/chats/assistantFeedbackRecordsClient.test.ts
bun run --cwd apps/cms-leverage test:integration -- tests/int/payloadConfig.int.spec.ts tests/int/routes.int.spec.ts
bun run --cwd packages/cms-customer-contracts test
bun run --cwd apps/web test -- test/nuxt/ChatMessageFeedback.nuxt.spec.ts test/nuxt/UseChats.nuxt.spec.ts
bun run check-types
bun run test:local

The app-level API test script sets in-memory observability for API tests. If the local CMS integration command intentionally uses a non-test local database, set CMS_LEVERAGE_ALLOW_NON_TEST_DB_FOR_TESTS=true for that run only.

Run bun run test:api:chat:review when chat prompts, orchestration, evals, scorers, datasets, experiments, or model-selection behavior changed.

For browser automation, start with agent-browser skills get core, then use the repo login helper when an authenticated web session is needed: bun run agent-browser:login:web.

Troubleshooting

User expects review but UI says review is unavailable: confirm customer API has central handoff config and Leverage-hosted cms-leverage has the required CMS_LEVERAGE_FEEDBACK_JIRA_* Jira config.
Jira issue not created: inspect the feedback record’s reviewHandoffStatus, jiraLastAttemptAt, and jiraLastErrorCode; do not log or copy raw comment/message content into troubleshooting notes.
Jira token rotated after failures: run the handoff smoke first. Failed feedback records will not replay automatically until the planned retry command/job exists; track the affected safe feedback ids or counts without copying raw chat content.
Scoped Jira API token returns 401/403: confirm CMS_LEVERAGE_FEEDBACK_JIRA_API_BASE_URL is set to https://api.atlassian.com/ex/jira/<cloudId>, the Cloud ID is the /s/ UUID and not the /o/ Organization ID, and the token has the documented granular Jira scopes plus target project/space permissions.
Missing Jira handoff failure notification: this is currently a known gap. The canonical assistant-feedback docs require a future safe Leverage-operator notification path with reason codes and retry eligibility.
Customer handoff rejected by cms-leverage: verify deployment id, client certificate identity/status, trust bundle, timestamp skew, nonce replay status, body hash/signature, and deployment credential status. Do not paste raw comments or transcript text into troubleshooting notes.
Duplicate Jira issue concern: confirm whether the existing Jira issue is still open and whether the user removed feedback before submitting again. Repeat updates should not create a second active issue unless a completed issue received a rating flip or feedback was submitted again after removal.
User cannot find older notes: confirm the feedback record still exists for that message and that the active thumbs feedback menu shows Previous notes (<count>) when notes exist.
Feedback removed while handoff was pending: pending handoff should be cancelled. Existing Jira issues are intentionally not deleted automatically, and later review-bound feedback should create a new Jira task.
Reviewer wants to commit raw text in an eval: do not commit it. Rewrite the scenario synthetically or redact it unless a future accepted policy explicitly permits raw eval fixtures.
Ticket is too vague to become an eval: ask the reviewer to fill Expected Behavior To Preserve, Failure/Exemplar Type, Likely Eval Path, Synthetic Eval Prompt Draft, Acceptance Signal, and Privacy Class before implementation.