A prompt change is a code change with no type system and no test suite. Treating it as deployable without evaluation is how AI features regress silently.
Teams apply careful review to application code and ship prompt changes on vibes. The prompt has at least as much influence on behaviour, and unlike code it fails silently — no exception, no stack trace, just gradually worse answers.
The minimum viable evaluation
- A fixed set of representative inputs with known-good outputs, versioned alongside the prompt
- Deterministic assertions where possible — did it cite a source, did it stay within length, did it refuse when it should
- A stronger model grading the production model's output for the judgements that cannot be asserted mechanically
- The suite runs on every pull request that touches a prompt, a tool definition or a retrieval parameter
- Results recorded over time, so regression is visible as a trend rather than a surprise
On using a model as judge
It is imperfect and it is far better than nothing. Keep the rubric explicit, keep the judge model fixed while comparing runs, and spot-check its verdicts periodically against human judgement. A judge that drifts silently is worse than no judge at all.
Why this is a delivery concern
Without evaluation there is no way to distinguish an improvement from a regression, which means nobody can approve a prompt change with confidence. Teams respond by freezing prompts. The evaluation suite is what makes the feature safe to keep improving.