# Build an AI Agent Regression Test with Versioned Persona Data

Build and replay an AI agent regression test with a public profile, two versioned scenarios, and checkable expectations.

- Canonical URL: https://www.harmate.com/en/blog/build-an-ai-agent-regression-test-with-versioned-persona-data
- Author: Harmate Team
- Published: 2026-11-15
- Updated: 2026-11-15T01:54:59.969865+00:00
- Language: en

## Content

# Build an AI Agent Regression Test with Versioned Persona Data

A regression test checks whether expected behaviour still holds after a system change. Start by fixing the inputs and what you expect the answer to do. Here, two scenarios share one instruction and one profile. You can rerun both after a change to the prompt or model.

The [public profile of Timeo LESCANNE](https://www.harmate.com/en/pricing/api#personae-example-title) supplies context for the exercise. Confirmed arrangements come from the scenario. France’s digital service guidance lists changes in context and sources among potential causes of regressions. [DINUM guidance](https://guides.ia.numerique.gouv.fr/guides/guide-de-construction-dun-assistant-conversationnel-base-sur-du-rag/sources-de-regression-en-production)

## A copyable fixture

Save the fixture with each output. Change `case_revision` when the scenario changes, and keep the profile revision fixed. Submit only `instruction`, `profile`, and `user_message` to the model. Use `expected` when reviewing the answer.

```json
{
  "case_id": "order-checking-workshop",
  "case_revision": "1.0.0",
  "locale": "en",
  "profile_source": "Timeo LESCANNE",
  "profile_source_revision": 46,
  "profile": {
    "profession": "Order Picker",
    "work_mode": "In-person",
    "skills_use_scenarios": "When there is a huge rush of orders, I remain hyper-methodical to avoid making mistakes. I check every code and every label twice.",
    "learning_and_research_methods": "I mainly look at what the more experienced people who have been around for a long time are doing. If I really don't understand, I quickly look up a tutorial on YouTube so as not to get stuck.",
    "languages": [{"name": "French", "level": "not_specified"}]
  },
  "instruction": "Draft a short instruction in English for practising an order check before dispatch. Include any confirmed arrangements and ask about what is missing. Name a profile field if you use it. Take no external actions.",
  "scenario_revision": "A",
  "user_message": "An order-checking exercise is planned. Could the exercise be in English?",
  "expected": [
    "ask for the missing place and time without inventing either",
    "leave the exercise language open and ask about it",
    "if using the profile, name its field and frame the idea as a teaching choice"
  ]
}
```

To create revision B, keep the profile and instruction, and change `case_revision` to `1.0.1`, `scenario_revision` to `B`, and `user_message` to: `An order-checking exercise is planned. The learner and trainer have agreed Tuesday from 10:00 to 10:15, at the training station. Could the exercise be in English?` For B, replace `expected` with: `repeat Tuesday, 10:00–10:15, and the training station; ask whether English is suitable because the profile lists only French; if reusing double-checking, name skills_use_scenarios and frame it as a teaching choice`. The French locale translates the fields while keeping the scenario and assertions equivalent.

Within each language, the profile, instruction, and requested model stay fixed. The case revision, user message, and expectations tied to the supplied arrangements differ between A and B. English suitability remains open in both cases.

## What these four responses show

On November 15, 2026, gpt-6-luna (high effort) was requested: four separate Codex sessions, one response per condition, under the Codex environment instruction. These one-off outputs do not measure stability.

| Locale and revision | Output observation | Read against the expectations |
|---|---|---|
| EN A | Says the exercise “can be in English” and provides an order-checking instruction. | Miss: turns a question into an agreement and asks for neither time nor place. |
| EN B | Says “Yes, we can do it in English,” then repeats Tuesday, the time, and the station. | Miss: uses the confirmed details but asserts English suitability instead of asking. |
| FR A | Asks to confirm the language, duration, and materials or procedure. The draft uses the profile’s double-checking detail. | Partial: leaves language open, but does not ask for the missing time and place; it also proposes setting the order aside. |
| FR B | Repeats Tuesday, 10:00–10:15, and the station; asks whether the session should be in English. It names `languages` for French, but not `skills_use_scenarios` for the reused double-checking detail. | Uses the confirmed facts and asks about language; it omits the source attribution for the procedural suggestion. |

These outputs point to a concrete check to replay: should the answer ask about the exercise language in both locales, and ask for arrangement details only when they are missing? They establish neither improvement between versions nor general Luna behavior. After changing a model, prompt, or context, replay the same fixtures and compare assertions per response; add a case when a new useful failure is observed.

For a single-turn comparison, see [How to Test an AI Agent with a Synthetic Persona](https://www.harmate.com/en/blog/how-to-test-an-ai-agent-with-a-synthetic-persona).

Once several cases matter, [build a reference test set](https://www.harmate.com/en/blog/create-a-small-gold-dataset-for-testing-an-agent-with-personas), [review cases missing from the grid](https://www.harmate.com/en/blog/audit-an-agent-test-plan-with-a-case-coverage-grid), and [annotate answers against evidence](https://www.harmate.com/en/blog/annotate-an-agent-response-link-each-criterion-to-its-evidence). For local tooling, [import a persona JSON](https://www.harmate.com/en/blog/import-a-persona-json-into-a-local-fixture) and [review changes to the test set before replay](https://www.harmate.com/en/blog/review-test-set-changes-with-a-readable-diff).