An agent can sound certain while the software is wrong. It can also sound hesitant after producing a correct, well-tested patch. Confidence is a property of language; completion is a claim about the environment.
Verification-first engineering starts from that separation. Before execution, define what evidence would make the claim true. After execution, grade the outcome—not the persuasiveness of the final message.
Feedback is not yet verification
The six-layer harness model separates feedback from verification for a reason. Feedback reports what an action returned: exit code zero, a rendered page, a changed file, a database row, or a screenshot. Verification compares those observations with an acceptance criterion.
A green command may prove only that the command completed. It does not automatically prove that the right tests ran, the user-visible behavior changed, unrelated behavior remained intact, or a policy boundary was preserved. The verification question is always narrower and more explicit: “What claim does this evidence support?”
Anthropic’s guide to evaluating AI agents distinguishes the transcript—the record of a trial—from the outcome—the final state in the environment. A booking agent may say a flight was booked; the outcome check asks whether the reservation exists. For a coding agent, the equivalent is checking the repository and running system, not accepting the handoff prose.
Define evidence before the run
An acceptance criterion should describe observable state. “Improve the form” is hard to verify. “Submitting an invalid email shows the inline error, keeps the entered value, moves focus to the field, and sends no request” gives the harness multiple testable claims.
Good criteria cover three dimensions:
- Do not break things: existing behavior and invariants remain intact.
- Do what was requested: the target outcome exists in the environment.
- Do it well: accessibility, performance, security, maintainability, or product taste meet the relevant bar.
The last dimension may require a rubric or human review, but it should still be stated before the patch is judged. An undefined standard invites the evaluator to reward whatever looks plausible after the fact.
| Claim | Strong evidence |
|---|---|
| Code integrity | Static checks and focused tests |
| User behavior | Runtime and end-to-end checks |
| Product quality | Calibrated rubric or human judgment |
Build a layered verification pyramid
Use deterministic checks wherever they can express the requirement. Static analysis, types, formatting, unit tests, schema checks, and security rules are fast, cheap, reproducible, and easy to debug. They form the broad base.
Integration and contract tests verify boundaries: database behavior, API compatibility, migrations, service wiring, and dependency assumptions. Behavior, end-to-end, and visual checks move closer to the user outcome. They cost more and may be less stable, so apply them to critical paths rather than every permutation.
Human judgment belongs at the top for questions that remain contextual: whether a design feels coherent, a trade-off matches product strategy, a destructive action is acceptable, or a subtle refactor will remain understandable. The goal is not to eliminate judgment. It is to spend it only where cheaper evidence cannot decide.
This layered approach resembles the multi-method strategy in Anthropic’s agent-eval work: code-based graders for objective conditions, model-based graders for structured nuance, and humans for calibration and expert judgment. No single grader catches every failure.
Package evidence for a decision
A useful handoff does not dump the entire transcript. It compresses the trajectory into an evidence packet:
- the requested outcome and constraints;
- the files or state that changed;
- the focused checks used during iteration;
- the broader checks run before completion;
- the relevant results, screenshots, or metrics;
- unresolved risks, skipped checks, and approvals.
Compression must retain provenance. “Tests pass” is weak; “52 tests passed under command X at revision Y” is reviewable. “UI looks good” is weak; named viewport screenshots and an overflow measurement are stronger. A failing check should remain visible rather than being buried beneath a successful retry.
The packet supports a decision: accept, iterate, escalate, or reject.
Grade outcomes without overfitting the path
Agents can reach a valid solution through steps an evaluator did not anticipate. Requiring one exact tool sequence often creates brittle grading. Prefer checking the final environment, required safety properties, and a few path constraints that truly matter—such as obtaining approval before a production change.
SWE-bench formalized outcome-oriented coding tasks by giving a system a real issue and repository, then evaluating the resulting patch with tests. The original SWE-bench paper helped make repository-level outcomes a central unit of coding-agent evaluation.
But tests are still an approximation. They may be incomplete, leaked, misconfigured, or too narrow. Recent work on contextual agentic rubrics explores repository-specific criteria that can flag issues beyond executable tests. The important design principle is triangulation: combine independent signals whose blind spots do not fully overlap.
Inspect the trajectory as well as the outcome
A passing outcome is necessary, but some tasks also require process evidence. The agent may have changed unrelated files, bypassed a test, used an unsafe permission, exposed a secret, or relied on state that another run cannot reproduce.
Trace review asks different questions:
- Did the agent operate within the stated scope?
- Were risky actions authorized before execution?
- Did observations actually justify the next step?
- Were failures diagnosed or merely retried until green?
- Can another engineer reproduce the checks?
- Does the stop reason match the remaining state?
A useful failure can have a wrong outcome and a strong trace: the task remains incomplete, but the system has a precise diagnosis and no hidden damage. Conversely, a correct outcome with an opaque trace should not automatically earn trust. Reliability means the result can be repeated and explained.
Use a verification-first protocol
A lightweight protocol can fit almost every coding-agent task.
- Define acceptance. Convert intent into observable claims and explicit non-goals.
- Run layered checks. Start focused, then broaden in proportion to the risk.
- Inspect evidence. Confirm the check measured the intended behavior.
- Challenge blind spots. Look for missing cases, grader bugs, and accidental shortcuts.
- Record the decision. Preserve what passed, what did not run, and why the task stopped.
A reference solution or known-good example can validate that the task and grader are themselves solvable. Balanced cases should test when behavior must occur and when it must not. Model-based graders should be calibrated against human judgment, not treated as objective truth.
Verification is the feedback loop for the harness
Verification does more than accept or reject one patch. Failed criteria reveal which part of the harness needs improvement: missing repository knowledge, an ambiguous tool, a runtime mismatch, weak observation formatting, or an incomplete test surface.
That makes evidence the learning signal for harness engineering. Confidence may change with wording. Evidence changes the system.
The final article turns this operating model into a role: the skills, ownership boundaries, maturity ladder, and first 90 days of a Harness Engineer.
Sources
- Anthropic — Demystifying evals for AI agents
- OpenAI — Harness engineering
- Jimenez et al. — SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Raghavendra et al. — Agentic Rubrics as Contextual Verifiers for SWE Agents
