How we cut LLM costs by 99% without hurting quality
Most AI features are expensive because nobody measured them. Here is the routing, caching and prompt work that took one production workload from thousands to tens of dollars a month.
When a team ships its first feature on top of a large language model, the bill is rarely the first concern. It becomes the first concern about two months later. The workload we describe here processed long documents for a SaaS product and cost several thousand dollars a month at launch. Six weeks later it cost under fifty. Quality, measured on a fixed evaluation set, went up.
1. Measure before optimising
The first week produced no code changes at all. We logged every call with its token counts, model, latency and the identifier of the feature that triggered it. Two findings paid for the whole project: 38% of calls were retries of requests that had already succeeded, and one background job re-summarised documents that had not changed.
2. Route by difficulty, not by habit
Every request had been sent to the largest available model. We classified requests into three tiers using cheap heuristics (document length, language, whether structured output was required) and sent each tier to the smallest model that passed the evaluation set for that tier. Roughly 70% of traffic moved to models that cost 10 to 20 times less per token.
3. Cache aggressively, at two levels
- Exact-match cache on the normalised prompt. Boring, and it removed the retry waste immediately.
- Prompt caching offered by the providers for the long, static part of the prompt (instructions, schemas, few-shot examples). On this workload the static prefix was 80% of the input tokens.
4. Shrink the prompt
The system prompt had grown to 3,000 tokens of accumulated instructions. We rewrote it to 600, moved examples into a retrieval step that only injects the two most relevant ones, and switched to structured outputs so the model stopped producing explanatory prose we discarded anyway.
5. Batch what does not need to be interactive
Nightly re-processing moved to the providers' batch APIs at half price and no longer competed with interactive traffic for rate limits.
What we kept
An evaluation set of 400 real, anonymised examples with expected outputs, run on every change. Cost work without an evaluation set is guessing. The set is also what let us say "quality went up" with a straight face.
The playbook in one list
- Log tokens, model and feature per call.
- Build an evaluation set from real traffic.
- Kill duplicate and unchanged work.
- Route by tier to the smallest passing model.
- Cache the static prompt prefix.
- Cut the prompt, use structured outputs.
- Batch the offline work.
If your AI feature has a budget line that makes someone wince, this is usually two to four weeks of work. Talk to us if you would like help doing it.
