Evals as contracts: how to know your AI system is working

Here is a meeting I have sat in more than once. An AI feature has been live for nine months. The team shows a slide: 94% accuracy on the eval set. Someone from the business asks the question nobody has an answer to: "What happened to the other 6%?"

Silence. The evals ran once, before launch. The number went on a slide. Nobody has looked at the failures since, and nobody was ever asked to.

The single biggest mindset shift I push on clients is this: stop thinking of evals as tests and start thinking of them as contracts.

Tests are things you run and hope they pass. Contracts are things you commit to. They define what the system is and is not allowed to do, and everyone (the engineers, the business owner, the compliance team) signs before anything ships.

The problem with evals as tests

When evals are framed as tests, they get treated like unit tests: nice to have, occasionally run, mostly ignored once the feature "works". I have seen teams build elaborate eval harnesses that run once, produce a 94% accuracy number, and then collect dust while the production system drifts unmonitored for nine months.

The root cause is that nobody owns the 6%. Tests leave the failure cases in a grey zone. They exist, they are measured, but there is no process for deciding what to do about them, and no consequence when they grow.

A single accuracy number makes this worse. It hides which failures happened, to whom, and whether any of them were the kind you cannot take back.

Contracts name the failures

An eval as contract is different. It says: for this system, these categories of input must produce outputs with these specific properties. Here is what happens when they do not.

A concrete example from a healthcare client:

• Input category: "patient asks about medication interaction"
• Required output property: must include the phrase "consult your doctor" OR route to human escalation
• Measurement window: 7 days, rolling
• Threshold: 0.1% violation rate
• Automatic action: if the threshold is breached, the system is automatically taken offline and the on-call engineer is paged

That is not a test. That is a service-level contract. It tells you what the system is, what the business tolerates, and what happens when reality diverges. Notice what it does not contain: a general accuracy score, a vibe, or a hope.

How to write one

Start with the three failure modes you would be fired for. Not the ones you are proudest of catching. The ones that would end your career if they happened in production and you had not seen them coming.

Write them down. For each, define:

1. The exact input shape that triggers the risk
2. The exact output property that must hold
3. The measurement window and threshold
4. The automatic action when the threshold is breached

That is a contract. It fits on one page. It gets signed.

Most teams find that the three "fired for" failures are not the ones their eval set was measuring. That discovery alone pays for the exercise.

The one-page template

Copy this into a document and fill it in. One row per failure mode; three rows to start.

SYSTEM: [name and one-line purpose]
OWNER: [one person, by name]

FAILURE 1: [what goes wrong, in plain words]
  Trigger input:      [the shape of input where it can happen]
  Required property:  [what the output must always / never contain or do]
  How we measure:     [automated check, sampled human review, or both]
  Window:             [e.g. 7 days rolling / per 1,000 requests]
  Threshold:          [e.g. 0.1% violations]
  Automatic action:   [e.g. route to human / take offline / page on-call]
  Reviewed by:        [who looks at every violation, and how often]

FAILURE 2: ...
FAILURE 3: ...

SIGNED: engineering ____  business owner ____  risk / compliance ____
NEXT REVIEW: [date, at most 90 days out]

Two rules for filling it in. First, "required property" must be checkable by a machine or by a person with a checklist, not by "someone senior would know". Second, "automatic action" must be something that happens without a meeting. If the action is "we discuss it", it is not a contract.

Setting thresholds without guessing

People freeze on the threshold. Here is how to pick one in ten minutes.

• Ask whether the failure is reversible. A wrong draft that a human reads before sending is reversible. Advice that a patient acts on is not. Irreversible failures get thresholds near zero and a stop-the-system action. Reversible ones can tolerate a rate and a review.
• Ask what the human process tolerated before. If the manual version of this work had an error rate nobody measured, measure it for a week. The contract should be at least as strict as the humans were, and you now have a number instead of an argument.
• Size the window to the volume. Ten requests a day cannot support a 0.1% threshold on a 7-day window; you would need months to see a single violation. Widen the window or count in requests ("per 1,000") instead of days.
• Start stricter than you think and loosen with evidence. A contract that trips too often is a good problem: it means you are looking. A contract that never trips is usually one nobody wired up.

Who signs, and what happens on Monday

A contract without signatures is a document. Three people sign: the engineer who wires the checks, the business owner who accepts the threshold, and whoever answers for risk or compliance. The act of signing changes the conversation. The business owner can no longer say "the AI made a mistake"; they agreed to the rate. The engineer can no longer say "the model drifted"; the automatic action was theirs to build.

Then there is a weekly ritual, and it is short. Every Monday, the owner opens the violations from the last window, reads five of them, and answers three questions: are these the failures we named, is the threshold still right, and did the automatic action fire when it should have. Fifteen minutes. If the meeting takes longer, the contract is too vague.

What good looks like

You will know the contract is working when:

• Anyone on the team can name the three failures without looking.
• A violation has fired the automatic action at least once, and nothing dramatic happened, because the action was designed for it.
• The threshold has been changed at least once, with a reason written next to it.
• The business owner refers to "our 0.1%" as their number, not engineering's.
• New features get a row in the contract before they get a launch date.

Common mistakes

• Reporting one accuracy number. It hides the failures that matter behind the ones that do not.
• Evals that nobody owns. If the owner field says "the team", it says nobody.
• Thresholds set by feel. Use reversibility and the old human error rate.
• Automatic actions that are actually meetings.
• Test sets that are too large to maintain and too random to matter. Fifty examples of the three failures beat five thousand examples of everything.

Why this matters for consultants

If you build AI for clients, the contract is the deliverable. Not the model. Not the prompts. Not the pipeline. The contract is what the client is buying, because the contract is what lets them sleep at night, and what lets them say to their own board exactly what the system will and will not do.

I used to ship my clients trained models and documentation. Now I ship them contracts and the code that enforces them. The second delivery is worth about 4× the first, and my clients renew.

Do this Monday

• Write down the three failures you would be fired for. Ten minutes, no laptop.
• Fill in the template for those three. Leave the threshold blank if you must; fill in everything else.
• Measure the current rate for one week, by hand if necessary.
• Set the threshold, wire one automatic action, and get three signatures.
• Put fifteen minutes in the calendar for next Monday. That meeting is the contract.