# Create a Small Gold Dataset for Testing an Agent with Personas A JSONL format, checkable criteria, and clear limits for building a small reference dataset for agent testing. - Canonical URL: https://www.harmate.com/en/blog/create-a-small-gold-dataset-for-testing-an-agent-with-personas - Author: Harmate Team - Published: 2026-11-15 - Updated: 2026-11-15T01:52:04.497147+00:00 - Language: en ## Content # Create a Small Gold Dataset for Testing an Agent with Personas A gold dataset is a set of cases a team can reuse to inspect an agent’s behavior. Each record links a fictional profile, a task, a response, and a few observable criteria. The aim is to make examples repeatable and open to review, not to declare that an agent is generally reliable. ## Decide what the dataset can check Start with a narrow question. For example: “Does the agent respect known facts and flag missing information when suggesting a next step?” A case should isolate a useful condition: an explicitly stated preference, a time constraint, a missing fact, or conflicting information. Avoid asking one case to check tone, accuracy, safety, relevance, and personalization all at once. Too many criteria make the result hard to interpret. Each criterion should be reviewable in the response or checked against the case itself. ## Store examples in a reusable format A simple JSONL file works well: one JSON object per line. Keep fields stable, such as `case_id`, `version`, `provenance`, `profile_facts`, `task`, `response`, and `criteria`. Each criterion can have an ID and an expected result such as “satisfied,” “not satisfied,” or “indeterminate.” Here are three fictional cases in JSONL format, one object per line. They match the English structural proof and retain its IDs and criteria: ```json {"case_id":"route-01","version":"1.0","provenance":"original fictional fixture","profile_facts":["Mina prefers written instructions.","Mina has 20 minutes available."],"task":"Suggest a first step for preparing a short team briefing.","response":"Draft a one-page written outline, then use the remaining time to rehearse the opening.","criteria":[{"id":"uses_stated_preference","expected":"satisfied"},{"id":"respects_time_limit","expected":"satisfied"},{"id":"does_not_invent_role_or_skill","expected":"satisfied"}]} {"case_id":"route-02","version":"1.0","provenance":"original fictional fixture","profile_facts":["No preferred format was stated.","The meeting is tomorrow."],"task":"Suggest how to prepare a project update.","response":"Ask whether the team wants a written summary or a spoken update; meanwhile, collect the current status and open decisions.","criteria":[{"id":"marks_missing_preference","expected":"satisfied"},{"id":"uses_known_deadline","expected":"satisfied"},{"id":"does_not_invent_project_details","expected":"satisfied"}]} {"case_id":"route-03","version":"1.0","provenance":"original fictional fixture","profile_facts":["Omar says he cannot attend on Thursday.","The organizer's note says Omar confirmed Thursday."],"task":"Draft a scheduling reply.","response":"I have conflicting information about Thursday. Could you confirm whether Omar can attend before I send the invitation?","criteria":[{"id":"surfaces_conflict","expected":"satisfied"},{"id":"avoids_choosing_a_side","expected":"satisfied"},{"id":"does_not_send_invitation","expected":"satisfied"}]} ``` The profiles, tasks, and responses were created to demonstrate the structure. They do not come from a study or a persona library. A small validator can check that required fields exist, IDs are unique, and every criterion has an ID. ## Evolve the dataset without losing its history Add a case when it covers a decision or condition the existing examples cannot examine. Keep IDs stable, and increment the version when a change alters the meaning of a case or its criteria. Record why the change was made so results from different runs are not compared as if they used the same reference. During a review, the team can compare the observed response with the criteria and record disagreements. An “indeterminate” result means the case lacks a rule or evidence; it does not need to be forced into a pass or fail. To compare two agent versions chronologically, keep the same case ID, exact instruction text, run date, and raw response. As an illustrative format—not an actual execution—suppose that on `route-02` one version advises preparing “the final finance report,” although no project status was supplied, while a later version asks about the update format and gathers known decisions. The first response fails `does_not_invent_project_details`; the second satisfies it. This single case describes a response difference on a designed example; it does not measure a success rate or the general effect of a change. ## Keep conclusions at the right scale A team-authored dataset helps repeat the same examples and make discussions more consistent. It does not show that the cases represent users, cover every situation, or yield a general success rate for a model. Performance claims require a defined population of relevant cases, suitable data, and a justified evaluation method. Treat these records as controlled fixtures: they document what you chose to examine, under which conditions, and against which criteria. They are a working aid, not a verdict on an agent’s overall quality. For a versioned regression case focused on an arrangement detail becoming confirmed, see [Build an AI Agent Regression Test with Versioned Persona Data](https://www.harmate.com/en/blog/build-an-ai-agent-regression-test-with-versioned-persona-data). The [three-part persona template](https://www.harmate.com/en/blog/synthetic-persona-template-a-minimal-card-for-testing-an-ai-assistant) keeps profile preferences, scenario facts, and response rules separate during review.