Most AI features bill far more than they need to, because every request goes to the largest available model regardless of difficulty.
The first production bill is usually the moment anyone looks at token economics. The pattern is nearly always the same: a single large model handling everything from 'what are your opening hours' to genuine multi-step reasoning, at identical cost per call.
Where the money actually goes
- Uniform model choice — routing easy requests to a smaller model is the largest single saving available
- Unbounded context — passing entire conversation history every turn, when a summary plus recent turns performs the same
- Retrieval sprawl — stuffing twenty chunks into the prompt because five felt too few to trust
- No caching — identical questions re-answered from scratch, forever
- No ceiling — nothing stopping a loop or an abusive client from running up spend overnight
The order to fix them
Put a hard global budget in place first. It is the least interesting change and the only one that limits your worst case. Then measure which requests are genuinely hard — usually a minority — and route the rest to something cheaper. Then trim context.
Measure quality before and after with a fixed evaluation set. Cost work without measurement is how a feature quietly degrades until someone complains three months later.