On a Friday night last December, our triage agent decided that a particularly polite customer was actually a phishing attempt — and politely told them so. Five minutes later, our oncall paged. Twenty minutes after that, we'd rolled back the prompt change that triggered it. Forty minutes after that, we were arguing in Slack about how to make sure this never happened again.
We had a prompt registry. We had reviewers. We had logs. What we didn't have was a way to know — before the change shipped — that the new prompt would refuse the wrong people. We were treating prompts like configuration: small, plain-text changes, version-controlled in Git, reviewed by a teammate, deployed in seconds. The Friday outage convinced us that was wrong.
Prompts aren't config. They're behavior.
When you change a YAML key, you can predict the result. Either the new value parses or it doesn't, and either it changes the behavior you intended or it doesn't. Prompts are different. A two-word change can shift the model's tone, its willingness to refuse, its tendency to hedge — across a thousand inputs, in subtle ways, that no human reviewer can hold in their head. Treating that surface area like a config file was, in retrospect, the kind of category error that only feels obvious after it bites you.
“We were reviewing prompts the way you'd review a CSS change. The model treated them like a rewrite of our refund policy.”
What we built instead.
We replaced the registry with versioned eval datasets — small bundles of inputs and expected outputs, each tied to a specific prompt revision. Instead of approving a prompt change in code review, you approve it by running the dataset and showing that the new behavior matches the new intent. The diff isn't text-against-text anymore. It's behavior-against-behavior.
We replaced the registry with versioned eval datasets — small bundles of inputs and expected outputs, each tied to a specific prompt revision. Instead of approving a prompt change in code review, you approve it by running the dataset and showing that the new behavior matches the new intent. The diff isn't text-against-text anymore. It's behavior-against-behavior.
The dataset lives next to the prompt in the repo. CI runs it on every change. If a contributor ships a prompt that flips refusals from 47 to 35, the build is red — long before the reviewer notices, and weeks before a customer hits it.
What we learned.
- Datasets need to be small enough to evolve. If your eval set takes 90 minutes to run, no one runs it locally. We aim for 30-60 examples per behavior dimension, and we ruthlessly delete examples that no longer surface real failures.
- The hardest examples come from production. Synthetic eval sets feel safe but rarely catch the failures that matter. Pull a fresh slice from production traces every Friday — anonymized, deduplicated, labeled by whoever is on triage that week.
- Pass/fail isn't the only useful signal. We track 'rate of disagreement with the previous version' as a secondary metric. A prompt that suddenly disagrees with itself on 30% of inputs is interesting, even if all of those answers technically pass.
Six months later, we haven't had another Friday-night incident. We've shipped fewer prompt changes overall — partly because the eval bar is higher, partly because we now notice that a lot of the prompt changes we used to ship were marginal. The ones that pass our datasets ship faster, with more confidence, and stay shipped longer.
Production AI doesn't need a new methodology. It mostly needs the old ones — taken seriously.
