# Audit an Agent Test Plan with a Case Coverage Grid

A coverage grid for reviewing an agent test plan without mistaking an empty cell for a proven defect.

- Canonical URL: https://www.harmate.com/en/blog/audit-an-agent-test-plan-with-a-case-coverage-grid
- Author: Harmate Team
- Published: 2026-11-15
- Updated: 2026-11-15T01:52:07.957134+00:00
- Language: en

## Content

# Audit an Agent Test Plan with a Case Coverage Grid

A coverage grid shows which combinations a test plan includes and which have no example yet. By itself, it does not say that an agent will fail in an empty cell. Its purpose is to make test choices visible so the team can decide what to add, what to leave out, and why.

## Cross tasks with conditions

First list the tasks you want to inspect: summarize a request, suggest a next step, or draft a message. Across the columns, note conditions that can change what the agent knows: a stated fact, missing information, conflicting facts, or an explicit constraint.

Assign each existing case to one cell or to several cells if the scenario truly checks those conditions. A reference such as `route-03` makes the example easy to find; an empty cell simply means no case is assigned there in this version of the plan.

Here is a small, entirely designed example:

| Task | Stated fact | Missing information | Conflicting facts | Explicit constraint |
| --- | --- | --- | --- | --- |
| Summarize a request | route-01 | route-02 | — | — |
| Suggest a next step | route-01 | route-02 | route-03 | — |
| Draft a message | — | — | route-03 | route-01 |

In this grid, 7 of 12 cells have an assigned case. That calculation describes only the grid shown here. The five empty cells do not prove an agent defect or a meaningful gap: some combinations may be out of scope or low value.

To make the assignments auditable by hand, the three IDs refer to these fictional examples: `route-01` is a team-briefing preparation with a written-format preference and a 20-minute constraint; `route-02` is a project-update preparation with no stated format preference and a meeting tomorrow; `route-03` is a scheduling reply to conflicting information about Omar’s availability on Thursday. Assign a case to a cell only when its content actually addresses that condition.

## Review the empty cells

For each empty cell, ask three questions: Is the situation part of the intended use? Does a neighboring case already cover the same behavior without ambiguity? Could this test change a design or release decision? If not, record why the cell is excluded. If yes, draft a minimal candidate that isolates the missing condition.

Not every combination matters equally. Do not fill the grid just to raise a percentage. An additional case is useful when it lets you observe a distinct behavior, not merely because it makes the table look more complete.

## Automate discovery and keep the decision with the team

A CSV with one column per condition may be enough. A short script can count assigned cells and list empty ones. In the example above, it reports five cells without a case. This mechanical check cannot tell whether those absences matter, whether an example is classified correctly, or whether the criteria are relevant. The team still needs to review the plan and document its choices.

After adding a case, rerun the script and keep the grid version with the test set. The resulting number is a local indicator for this list of tasks and conditions; it cannot automatically be compared with another product or version.

## Interpret without overclaiming

A grid supports a statement such as: “We assigned cases to these cells and not to those.” It does not show that coverage is complete, representative, or predictive of real-world use. Its value is the structured discussion it enables before deciding which examples are worth writing.

For an example of a versioned regression fixture, see [Build an AI Agent Regression Test with Versioned Persona Data](https://www.harmate.com/en/blog/build-an-ai-agent-regression-test-with-versioned-persona-data).

To decide whether a new case changes the decision under test, use the [test profile comparison](https://www.harmate.com/en/blog/choose-a-test-persona-for-the-decision-you-need-to-check).