The Minimum Eval Harness for a Production AI Agent (Most Teams Skip This)
A minimum eval harness is five pieces: a small curated set of 15-50 real tasks with expected outcomes, offline evals in CI on every change, a few automatable checks (structured-output match, an LLM-as-judge rubric, tool-call correctness), online evals on a sample of production traffic, and a loop where failures become new cases. No platform required — a test file and your traces beat dashboards with no grades.
The 2026 gap: teams watch their agents but can’t grade them
Here’s the gap in one breath: 89% of teams have agent observability, but only 52.4% run offline evals and 37.3% run online evals on real traffic (LangChain State of Agent Engineering 2026, 1,300+ professionals). Teams can watch their agents but can’t grade them. Below is the minimum eval harness to close that gap — startable this week, no platform.
LangChain’s State of Agent Engineering 2026 report surveyed more than 1,300 professionals, and one number pair jumped off the page for me: 89% have implemented observability for their agents, but only 52.4% run offline evals and just 37.3% run online evals on real traffic (that 37.3% is the all-respondents figure; it climbs to 44.8% among teams already in production).
Read that again. Nearly nine in ten teams can watch their agent. Barely a third can tell you whether what it did was actually good.
The rest of the report fills in the picture. 57.3% have agents in production, and about a third — 32% — cite quality as their top barrier to deploying more. So we have a field full of teams that put agents in front of customers, instrumented them beautifully, and then got stuck on quality precisely because they never built the thing that measures quality.
The framing I keep coming back to: observability tells you WHAT happened; evals tell you whether it was GOOD. Most teams have the first and skip the second.
My opinion, labeled as opinion: shipping an agent you can watch but can’t grade is shipping blind with a nice dashboard. The traces feel like progress. They’re not. They’re raw material. If you’ve already got an agent running in production, the gap below is probably your gap too.
Observability vs evals: what happened vs was it good
These two words get used interchangeably, and that confusion is exactly why teams skip the part that matters.
Observability is traces, logs, spans, latency, token cost. It answers what happened on this run. Your agent received a message, called three tools, spent 4,200 tokens, returned in 1.8 seconds. All true, all useful, all silent on the only question your customer cares about.
Evals are a graded verdict against expected behavior. They answer was that output correct. Did the agent pick the right tool, with the right arguments, and produce an answer that matches what a good answer looks like?
A trace showing a tool call fired tells you nothing about whether it was the RIGHT tool with the RIGHT arguments. This is the whole trap. The span is green, the latency is fine, and the agent still refunded the wrong customer.
That payments example is not hypothetical for me. A trace shows my agent called the refund tool and got a 200 back. Only an eval tells me it refunded the correct amount to the correct customer. Observability gives you the raw material — the traces. Evals turn a sample of those traces into scores. You need both, in that order.
Observability — what happened
- Traces, logs, spans
- Latency and token cost
- Which tools fired, in what order
- Silent on correctness — a green span can be a wrong refund
Evals — was it good
- A graded verdict vs expected behavior
- Right tool, right arguments?
- Output matches a good answer?
- Turns a sample of traces into scores you can gate on
How many eval cases do you need? Start with 15-50 real tasks
Do NOT start with a giant public benchmark. Start with 15 to 50 real tasks pulled from your own traffic.
Each case needs two things: an input and an expected outcome. The expected outcome can be an exact answer, an expected tool call, or a set of rubric criteria — pick whichever fits the task. Mine your production traces and support tickets for the cases that actually occur and, especially, the ones that broke. Those are gold; they already failed once.
LATAM tie, because this is where most teams quietly lose: for a Spanish support or payments agent, write the cases in the real Mexican-Spanish phrasing your customers use — “me cobraron de más”, “quiero mi reembolso”, “no me llegó la factura” — not translated English. An English benchmark will never catch how your customers actually ask. This is the same grounding discipline that keeps a chatbot from hallucinating: you verify against what real users actually say.
Opinion, labeled: 30 hand-picked real cases beat a 1,000-row benchmark you don’t understand. You can read 30 cases. You can’t read 1,000, which means you can’t trust the score they produce.
Why run offline evals in CI on every change?
Wire the eval set into CI so it runs on every prompt edit, model swap, or tool change. Assert task success, not vibes — a failing eval should fail the build like any other test.
This is the 52.4% bucket from the report, but the discipline that actually pays off is running it on EVERY change, not once at launch. A launch-day eval is a snapshot. A CI eval is a guardrail.
Here’s a minimal offline eval as a plain test file — no platform required:
# test_agent_evals.py — runs in CI on every change
import pytest
from my_agent import run_agent # your agent entrypoint
# 15-50 real cases mined from production. Start small.
EVAL_SET = [
{
"id": "refund_overcharge_es",
"input": "Me cobraron dos veces el mismo cargo, quiero mi reembolso",
"expected_tool": "create_refund",
"expected_args": {"reason": "duplicate_charge"},
},
{
"id": "invoice_missing_es",
"input": "No me llegó la factura de mi compra de ayer",
"expected_tool": "resend_invoice",
"expected_args": {},
},
]
@pytest.mark.parametrize("case", EVAL_SET, ids=lambda c: c["id"])
def test_agent_picks_right_tool(case):
result = run_agent(case["input"])
# tool-call correctness: right tool, right arguments
assert result.tool_name == case["expected_tool"]
for key, val in case["expected_args"].items():
assert result.tool_args.get(key) == valCatch regressions before they reach a customer. A model upgrade that improves one task can silently break three others — and the payments tie is sharp here: a prompt tweak that helps general chat can break the strict refund-eligibility logic. CI catches it; a dashboard does not.
What checks actually grade an agent? Structured match, LLM-judge, tool-call correctness
You need a small toolkit of grading mechanics. Three cover most cases.
Exact / structured-output match. Where the agent returns JSON, assert against a schema. Anthropic’s API does this with output_config.format using type: "json_schema". But be honest about what that buys you: structured outputs guarantee SHAPE, not TRUTH. The JSON will parse and match the schema — the value inside can still be wrong. So validate the VALUES too.
# Illustrative — verify field names against current Anthropic docs.
# output_config.format guarantees SHAPE, not TRUTH.
response = client.messages.create(
model="claude-opus-4-8",
max_tokens=1024,
messages=[{"role": "user", "content": case["input"]}],
output_config={
"format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"refund_amount_cents": {"type": "integer"},
"customer_id": {"type": "string"},
},
"required": ["refund_amount_cents", "customer_id"],
},
}
},
)
data = parse_structured(response)
# Shape is guaranteed by the schema. Now grade the VALUES:
assert data["refund_amount_cents"] == case["expected_amount_cents"]
assert data["customer_id"] == case["expected_customer_id"]LLM-as-judge rubric. For open-ended steps, score the output against a written rubric. Constrain the judge with output_config.format too, so you actually get back parseable JSON instead of free text:
JUDGE_RUBRIC = """
You are grading a Spanish-language payments support reply.
Score PASS only if ALL are true:
1. Cites the CORRECT refund policy (no invented fees).
2. Replies in natural Mexican Spanish, not translated English.
3. Does NOT promise a refund the policy does not allow.
"""
# Illustrative — verify field names against current Anthropic docs.
JUDGE_FORMAT = {
"format": {
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"verdict": {"type": "string", "enum": ["PASS", "FAIL"]},
"reason": {"type": "string"},
},
"required": ["verdict", "reason"],
},
}
}
def judge(agent_reply: str) -> dict:
resp = client.messages.create(
model="claude-opus-4-8",
max_tokens=512,
messages=[{"role": "user",
"content": JUDGE_RUBRIC + "\n\nReply:\n" + agent_reply}],
output_config=JUDGE_FORMAT, # guarantees {verdict, reason} shape
)
return parse_structured(resp) # {"verdict": ..., "reason": ...}Tool-call correctness. Assert the agent called the right tool with the right arguments — that’s the offline-eval check above. Note the separate strict: true flag applies to tool use, not to the structured-output format; don’t confuse the two.
Mix them per case: a structured field gets an exact check; a free-text explanation gets the judge. Honest limit: an LLM judge is itself fallible — spot-check its grades against your own judgment periodically, or you’re just trusting one model to grade another. If you need a refresher on what an AI agent actually is, start there and come back.
Why run online evals on a sample of production traffic?
This is the 37.3% most teams skip — it rises to 44.8% among teams already in production, and it’s the differentiator. Scoring REAL traffic, not just your curated set.
You don’t need to grade every request. Log traces, score a sample, alert on regressions. Even 5% of production traffic, scored daily with your rubric, will surface things your test set never imagined.
Online evals catch the drift offline tests miss: new customer phrasings, edge cases, a model provider quietly changing behavior under you. LATAM tie: real customers phrase things in ways your curated set never anticipated — sampling prod is how you find them, and then they become the next section’s new cases.
Payments tie, and this is the one that costs money: a quiet rise in wrong-amount refunds or failed fiscal lookups shows up in online evals before it shows up in churn. By the time it shows up in churn, you’ve already paid for it.
How do you make the harness compound? Close the loop — failures become new cases
Every failure caught in online evals — or in production — becomes a new case in your offline set. This is the flywheel, and it’s the difference between a harness that decays and one that compounds.
Over time, your eval set stops being a generic test suite and becomes a precise map of where YOUR agent actually breaks. Nobody else has that map. It’s built from your customers, your tools, your phrasing.
You can start this week, with no platform: a test file, your existing traces, an LLM-as-judge rubric, and a Spanish eval set on real customer phrasing. That’s it.
Opinion, labeled: a test file plus your traces beats “we have dashboards but no grades” every single time.
FAQ
Do I need a paid eval platform? No. Start with a test file and your traces. Platforms help at scale — at the start they’re a way to procrastinate.
How many eval cases is enough to start? 15 to 50 real cases beat a giant benchmark. Grow the set from real failures, not from a desire for a big number.
Do structured outputs make evals unnecessary? No. output_config.format with type: "json_schema" guarantees SHAPE, not TRUTH. The JSON validates; the refund amount inside can still be wrong. You still grade the values.
Can I just trust an LLM-as-judge? Mostly — but spot-check its grades against your own. It’s a tool to reduce risk, not eliminate it. A judge that drifts unnoticed is worse than no judge.
Why evals in Spanish? Because an English benchmark won’t catch how your Mexican customers actually phrase a refund request. Eval on real phrasing or you’re testing a customer you don’t have.
Isn’t observability enough? No. It tells you what happened, not whether it was correct. That exact gap is the whole point of this post.
- Curate 15-50 real casesPull from your traffic and tickets, in real customer Spanish
- Offline evals in CIRun on every prompt, model, or tool change — assert success
- Pick automatable checksStructured-output match, LLM-as-judge rubric, tool-call correctness
- Online evals on a sampleScore a slice of prod traffic; alert on regressions
- Close the loopEvery failure becomes a new offline case — the set compounds
Ship the harness this week
The five pieces are small enough to ship in a week: a curated set, CI offline evals, a few automatable checks, online sampling, and a feedback loop. None of it requires a platform.
Most teams that watch but don’t grade are stuck because they over-planned — they’re waiting for the right tool instead of writing the test file. Don’t be one of them. Open a test_agent_evals.py, paste in 20 real cases from your traces, and wire it into CI today.
If you build Spanish-first payments agents like I do, evals in your customers’ real phrasing aren’t a nice-to-have — they’re the edge. Want more on the agents you’re grading? Here are more AI-agent guides. And the deeper logic of why evals matter comes straight from the data: observability powers evaluation, but only if you actually build the evaluation half.