# Annotate an Agent Response: Link Each Criterion to Its Evidence

A traceable annotation method for reviewing an agent response without claiming to measure general evaluator accuracy.

- Canonical URL: https://www.harmate.com/en/blog/annotate-an-agent-response-link-each-criterion-to-its-evidence
- Author: Harmate Team
- Published: 2026-11-15
- Updated: 2026-11-15T01:52:11.465964+00:00
- Language: en

## Content

# Annotate an Agent Response: Link Each Criterion to Its Evidence

An evaluation grid is easier to review when each judgment points to two specific items: what the response says and what the case establishes. This traceability helps another reader understand why a criterion is marked satisfied, not satisfied, or indeterminate. It does not guarantee that the criterion is well chosen or that the evaluation is accurate in every context.

## Prepare a case and checkable criteria

Record the scenario facts and the instruction given to the agent. Then write criteria about observable behavior: Does it use an explicitly provided constraint? Does it present an assumption as fact? Does it ask for clarification when a decisive detail is missing? Avoid broad wording such as “good answer” unless you describe what should be observed.

Consider this fictional scenario: Lina has fifteen minutes before a client call; no presentation is mentioned. The agent replies: “Use the fifteen minutes to write three points and one question for the client. You can also review the slide deck.”

Separate two questions. First, what kind of support does each statement in the response have? Second, is the evaluation criterion satisfied? A support type describes the link between a statement and the case; a verdict describes the behavior targeted by a criterion. They are not interchangeable.

### Statement ledger

| Statement or interpretation | Support type | Case evidence | Reading |
| --- | --- | --- | --- |
| “Use the fifteen minutes” | Source paraphrase | Lina has fifteen minutes before the call. | The duration is restated in different words. |
| “write three points and one question” | Proposal | The task asks for a preparation step. | This is a suggested action, not a fact about Lina. |
| “Lina is rushed” | Reviewer inference | Lina has fifteen minutes before the call. | This interprets the time window; it is not response text or an established fact. |
| “review the slide deck” | Unsupported | No presentation is mentioned. | The case does not establish that one exists. |

An inference is a conclusion drawn from facts without being stated directly; mark whether it comes from the reviewer rather than from the response. “Unsupported” means the supplied case does not support the statement; it does not prove the statement false in the real world.

### Criterion ledger

| Criterion | Verdict | Statement reviewed | Rationale |
| --- | --- | --- | --- |
| Uses the known time constraint | Satisfied | “Use the fifteen minutes” | The response uses the stated duration. |
| Does not assume an unmentioned resource exists | Not satisfied | “review the slide deck” | The response assumes a deck is available. |
| Gives a concrete action | Satisfied | “write three points and one question” | The response gives an actionable step. |

The support types and verdicts above are designed annotations for this example, not a measure of evaluator accuracy. The first ledger classifies statements; the second checks behaviors against criteria.

| Criterion | Verdict | Response excerpt | Evidence in the case | Rationale |
| --- | --- | --- | --- | --- |
| Uses the known time constraint | Satisfied | “Use the fifteen minutes” | Lina has fifteen minutes before the call. | The suggestion uses the stated duration. |
| Does not assume an unmentioned resource exists | Not satisfied | “review the slide deck” | No presentation is mentioned. | The response assumes a deck is available. |
| Gives a concrete action | Satisfied | “write three points and one question” | The task asks for a preparation step. | The response gives an actionable step. |

The excerpts can be checked against the example and show how the two ledgers connect.

## Use “indeterminate” with an explicit rule

An indeterminate verdict is useful when the case lacks necessary evidence or the criterion allows multiple interpretations. Record what is missing: an operational definition, a scenario fact, or a decision rule. Do not replace the explanation with an average score that hides uncertainty.

If two reviewers disagree, keep both annotations and ask whether the disagreement comes from the criterion, the available facts, or the reading of the response. A rule can be revised after reviewing a response, but version the criterion and do not compare the two versions as if they were identical.

## Add a limited mechanical check

A validator can check whether each response excerpt appears in the response text and each cited evidence item appears among the case facts. This catches mistyped quotations. It cannot judge whether the criterion is relevant, the verdict is right, or the reasoning is sound.

Keep verdicts linked to their evidence and review cases in light of their purpose. This method makes a local review more transparent and open to challenge; it does not turn a fictional example into a benchmark or an annotation into a universal claim about agent performance.

For comparison with a versioned agent regression case, see [Build an AI Agent Regression Test with Versioned Persona Data](https://www.harmate.com/en/blog/build-an-ai-agent-regression-test-with-versioned-persona-data).

The [three-part persona template](https://www.harmate.com/en/blog/synthetic-persona-template-a-minimal-card-for-testing-an-ai-assistant) keeps profile preferences, scenario facts, and response rules separate during review.