Evaluating LLM output when there is no single right answer

Conventional tests assert equality. Language-model output is non-deterministic and legitimately variable in phrasing, so equality assertions either fail constantly or get deleted. What replaces them is a layered set of assertions, ordered from cheapest and most reliable to most expensive and least.

Four layers, in order of trustworthiness

Most of the value sits in the first two layers, and most teams skip straight to the fourth because it feels more sophisticated. Invert that: exhaust the deterministic assertions first, because they never produce a false verdict.

Structural assertions carry more weight than they appear to

For extraction, classification and tool-calling — which is most production AI work — validating the shape catches a large share of real failures before any judgement of quality is required.

// Validate the shape with the same schema the application uses, so the
// evaluation cannot pass output that production would reject.
const InvoiceSchema = z.object({
  invoiceNumber: z.string().min(1),
  issueDate: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
  totalAmount: z.number().positive(),
  currency: z.enum(["INR", "USD", "EUR", "GBP"]),
  lineItems: z.array(z.object({
    description: z.string().min(1),
    amount: z.number(),
  })).min(1),
});

// A structural failure is a hard fail — no judgement needed.
const parsed = InvoiceSchema.safeParse(output);
if (!parsed.success) {
  return { pass: false, reason: "schema", detail: parsed.error.issues };
}

// Internal consistency is still deterministic, and catches plausible
// output that is arithmetically wrong.
const summed = parsed.data.lineItems.reduce((t, i) => t + i.amount, 0);
const consistent = Math.abs(summed - parsed.data.totalAmount) < 0.01;

Score per field, not per response

A document extraction that gets nine of ten fields right is not a failure, and scoring it as one destroys your ability to see improvement. Field-level scoring shows which fields are weak, which is directly actionable.

That table tells you exactly where to spend effort, and a single aggregate pass rate would not. Line items are the problem; the date format is the second problem; nothing else needs attention.

Semantic assertions without overfitting to wording

For generated prose, exact matching is wrong and embedding similarity alone is unreliable — two paragraphs can be similar in embedding space while disagreeing on the fact that mattered. More robust approaches assert on specific content rather than global similarity.

• Assert required facts appear: the correct figure, the correct entity name, the cited source identifier
• Assert prohibited content does not appear: a competitor name, a guarantee, a recommendation you never make
• Assert refusal when the context genuinely does not support an answer — this is a test case, not an edge case
• Assert citations resolve to passages that were actually retrieved, rather than plausible-looking references

Model-as-judge, and where it misleads

Using a model to score output against a rubric scales where human review cannot. It is genuinely useful and it has well-documented biases that make it unsuitable as a gate on individual cases.

• Position bias: in pairwise comparisons, the option presented first tends to win — randomise order
• Length bias: longer answers are scored higher regardless of correctness
• Self-preference: models tend to favour output resembling their own style
• Poor calibration: absolute scores drift between runs; relative comparisons are more stable

Practical consequences: use judged scores as an aggregate trend across a whole evaluation set, prefer pairwise comparison to absolute scoring, use a different model as judge than the one under test, and never let a judged score alone block a release.

Running it in CI without it becoming unusable

Evaluations that take twenty minutes and cost real money on every pull request get disabled within a fortnight. Tier them by cost.

1. 01On every commit: structural assertions against recorded fixtures. Fast, free, deterministic.
2. 02On every pull request touching prompts, retrieval or model config: the deterministic-content layer on a 50-case subset.
3. 03Nightly: the full evaluation set including judged scores, with results tracked over time.
4. 04Before a model or prompt change ships: the full set, compared against the previous run rather than an absolute threshold.

Comparing against the previous run is the important detail. Absolute thresholds are arbitrary and get adjusted downward when inconvenient; a regression relative to last week is unambiguous.

The evaluation set is not the product

An evaluation set built up front reflects failures you anticipated. Production produces failures you did not. Sample real output continuously, and feed every genuine failure back into the set — which then permanently protects against that specific regression.

• Log inputs, outputs, retrieved context and confidence for a sampled percentage of production traffic
• Review a fixed number of sampled cases weekly — a small habit beats an occasional audit
• Route every confirmed failure into the evaluation set with its expected output
• Track the set's size as a health metric: an evaluation set that never grows is not being learned from

Over time this is what separates a feature that improves from one that merely ships. The harness stops being a gate and becomes the mechanism by which the system gets better.