# Review Test-Set Changes with a Readable Diff

A local Python script classifies test-set changes by case ID, without calling a model or API.

- Canonical URL: https://www.harmate.com/en/blog/review-test-set-changes-with-a-readable-diff
- Author: Harmate Team
- Published: 2026-11-15
- Updated: 2026-11-15T01:52:18.504406+00:00
- Language: en

## Content

# Review Test-Set Changes with a Readable Diff

A test set changes over time: a case can be added, corrected, or removed. If edits are mixed into a changed file without a comparison, a review can miss a moved instruction or a stricter expectation. A diff organized by case ID makes review more direct before you replay tests.

The example below is a fictional two-version set written for this guide. Each snapshot has a `set_id` and `set_revision`; every case has its own `case_id` and `case_revision`. Keep an ID stable while the case represents the same situation. When its input or expectations change, retain the ID and increment its revision. Give a genuinely different case a new ID.

```json
{
  "set_id": "library-assistant-smoke",
  "set_revision": "1.1",
  "cases": [
    {
      "case_id": "library-search-01",
      "case_revision": "2",
      "input": "Find opening hours.",
      "expected": ["ask for location if missing"]
    }
  ]
}
```

## Classify changes by ID

An ID only in the new version is added; one only in the old version is removed; an ID present in both is modified if any case field differs. Revision changes therefore appear as modifications and deserve review. The complete script below embeds both snapshots, checks their structure and duplicate IDs, and compares cases by ID. It requires only Python's standard library.

```python
import json

BEFORE = '''{"set_id":"library-assistant-smoke","set_revision":"1.0","cases":[{"case_id":"library-search-01","case_revision":"1","input":"Find opening hours.","expected":["ask for location if missing"]},{"case_id":"library-reservation-02","case_revision":"1","input":"Reserve this title.","expected":["do not claim a reservation"]}]}'''
AFTER = '''{"set_id":"library-assistant-smoke","set_revision":"1.1","cases":[{"case_id":"library-search-01","case_revision":"2","input":"Find opening hours.","expected":["ask for location if missing","distinguish confirmed hours from a guess"]},{"case_id":"library-reference-03","case_revision":"1","input":"What is the source?","expected":["name the source when provided"]}]}'''
FIELDS = {"case_id", "case_revision", "input", "expected"}

def load_set(text):
    data = json.loads(text)
    if not isinstance(data, dict) or set(data) != {"set_id", "set_revision", "cases"}:
        raise ValueError("invalid case-set shape")
    if not data["set_id"] or not data["set_revision"] or not isinstance(data["cases"], list):
        raise ValueError("invalid set ID, revision, or cases list")
    cases = {}
    for case in data["cases"]:
        if not isinstance(case, dict) or set(case) != FIELDS:
            raise ValueError("invalid case shape")
        case_id = case["case_id"]
        if not isinstance(case_id, str) or not case_id.strip() or case_id in cases:
            raise ValueError("blank or duplicate case ID")
        if not isinstance(case["case_revision"], str) or not case["case_revision"]:
            raise ValueError("invalid case revision")
        if not isinstance(case["input"], str) or not isinstance(case["expected"], list):
            raise ValueError("invalid input or expectations")
        cases[case_id] = case
    return data, cases

old_set, old = load_set(BEFORE)
new_set, new = load_set(AFTER)
if old_set["set_id"] != new_set["set_id"]:
    raise ValueError("cannot compare different set IDs")
old_ids, new_ids = set(old), set(new)
added = sorted(new_ids - old_ids)
removed = sorted(old_ids - new_ids)
modified = sorted(case_id for case_id in old_ids & new_ids if old[case_id] != new[case_id])
assert (added, removed, modified) == (["library-reference-03"],
    ["library-reservation-02"], ["library-search-01"])
print(f"Case set: {old_set['set_id']} ({old_set['set_revision']} -> {new_set['set_revision']})")
print("Added:", ", ".join(added) or "none")
print("Removed:", ", ".join(removed) or "none")
print("Modified:", ", ".join(modified) or "none")
print("Review rule: inspect each listed case before replay; no model was run.")
print("PASS expected stable-ID change classification")
```

Run it with `python3 review_case_set.py`. Expected output:

```text
Added: library-reference-03
Removed: library-reservation-02
Modified: library-search-01
Review rule: inspect each listed case before replay; no model was run.
```

## Review before replay

A diff cannot decide whether a change is appropriate. For each listed ID, inspect the exact fields, confirm why they changed, and review the expectations before running the affected cases. A removed case may represent a situation that is out of scope; an added case may cover a new decision. The diff measures neither overall coverage nor answer quality nor system improvement.

Keep the two snapshots you reviewed, the set revision, and the review notes. If you store outputs, link them to the case revision that produced them. A sound comparison tool should also reject duplicate IDs or a changed `set_id`, rather than silently assigning results to the wrong set.

This method helps inspect a multi-case set before replay. It complements an individual regression case: it makes changes to the set visible, but does not run scenarios or judge their results. The IDs and content here are fictional; adapt the format to your own tool and validate it with the data that belongs to it. For a single-case replay walkthrough, see [Build an AI Agent Regression Test with Versioned Persona Data](https://www.harmate.com/en/blog/build-an-ai-agent-regression-test-with-versioned-persona-data).

To decide whether a new case changes the decision under test, use the [test profile comparison](https://www.harmate.com/en/blog/choose-a-test-persona-for-the-decision-you-need-to-check).