Evals before vibes: release gates for LLM systems

Most LLM features ship on vibes. Someone tries a handful of prompts, the output looks good, and it goes live. Then a “small” prompt tweak three weeks later quietly breaks a case nobody re-checked — and in a regulated business, “it seemed fine” is not an answer you can give an auditor.

We treat an LLM system like any other production system: it doesn’t ship until it passes a gate, and the gate is built on measurement, not impressions.

Vibes don’t survive change

The core problem is that LLM behaviour is coupled in ways you can’t see. A change that improves one example often regresses ten others. Without a measurement you re-run on every change, you have no way to know whether you moved forward or sideways. Manual spot-checking doesn’t scale past the first week, and it never covers the cases that actually matter — the rare, high-consequence ones.

The offline eval suite is the unit test of an LLM system

Before a feature is “done,” we build an evaluation set: representative inputs paired with the properties a correct answer must have. Not always exact strings — usually a mix of checks:

• Programmatic checks where behaviour is binary: did it cite a source? did it refuse when the answer wasn’t in the provided documents? did it stay within scope?
• Reference checks where there’s a known-good answer or required facts.
• Rubric / LLM-as-judge grading where quality is fuzzier, with the judge itself validated against human labels so we trust its scores.

A single case looks like this:

- input: "What are our incident-notification timeframes under CPS 230?"
  must_cite: true
  must_contain: ["72 hours", "24 hours"]
  must_not: ["I think", "probably", "as an AI"]

The second phrase matters. An answer that mentions only the 72-hour rule for operational risk incidents reads as correct, but it is incomplete: CPS 230 also requires notice within 24 hours of a disruption to a critical operation outside tolerance. A check that looks for one phrase would pass the incomplete answer, so the case tests for every clock the question covers.

The suite runs in CI on every change to a prompt, a model, a retrieval setting, or a tool definition.

Gates, not dashboards

A dashboard you have to remember to look at is not a control. We turn the eval suite into a promotion gate: a change is allowed into production only if the aggregate score clears a threshold and there is zero regression on the “must never break” subset — the cases tied to compliance, safety, or money. Below the line, the release simply doesn’t ship. The gate is the same whether the change is a one-word prompt edit or a model upgrade.

Tracing so failures are explainable

Every request in production emits a trace: the retrieved context, the assembled prompt, the model and parameters, token counts, latency, cost, and — where applicable — the grader outcomes. When something goes wrong, you replay the exact request instead of guessing. For regulated buyers this is not just debugging; it is the audit trail that shows how a given answer was produced.

What “day one” actually means

The eval harness, the CI gate, and tracing go in during the first week of a build — not bolted on after the first incident. It feels slower for a fortnight and then it is permanently faster, because every subsequent change is safe to make. That is the whole point: discipline up front is what lets you move quickly later without breaking the things you can’t afford to break.