# The checklist we run before an AI feature ships

Naveen · 2026-08-06

Normal software either works or throws an error. AI features do something stranger: they work most of the time, and when they don't, they're confidently wrong in a full sentence.

After one launch where an assistant cheerfully told a user their (non-existent) meeting had been moved, we wrote a checklist. It's not a framework. It's a list in a doc, and the feature doesn't go out until each line is ticked.

## Before launch

**1. We have at least 30 real examples.**
Real inputs, written by real users or pulled from logs, with what a good answer looks like. Not inputs we invented at our desks. Ours always turn out too tidy.

**2. We ran them and read every output.**
A score is useful later. The first time, a person reads all of them. You find problems no metric would have caught, like the bot replying in English to a Tamil question.

**3. We included the awkward cases.**
Empty input. Very long input. A question the feature shouldn't answer. Someone trying to get it to ignore its instructions. A request that needs approval.

**4. We know what "wrong" costs.**
A bad summary is annoying. A wrong payment is a phone call from a lawyer. The riskier the action, the more of it goes through a human approval step, in code.

**5. There's a fallback.**
If the model times out or refuses, what does the user see? "Something went wrong" is not a fallback.

**6. We can see what it's doing in production.**
Inputs, outputs, tool calls, latency and cost per request, stored somewhere we can search. Without this you find out about problems from angry emails.

**7. We know the cost per task.**
Multiply it by expected usage. Then multiply that by three, because usage always grows faster than you planned.

## After launch

**8. Wrong answers become test cases.**
Every bad answer a user reports goes into the example set. The set grows. The next change is tested against all of it.

**9. Re-run when anything changes.**
New model, new prompt, new tool, new provider. Re-run the set and compare before switching. This is the line that lets us change models in an afternoon instead of a sprint.

## That's it

Nine lines. Nothing about which eval library to use, because it matters much less than actually having the examples and reading the outputs. A spreadsheet works. We started with one.

If you're about to launch something and want someone to try to break it first, email naveen@estreetz.com. We enjoy that part more than we should.