How to Reduce LLM Costs Without Regressing Answer Quality

Cutting an LLM bill is easy. Switch to a cheaper model and the number goes down the same day. The hard part is knowing what you gave up, because a quality loss in an AI system is often quiet. Nothing crashes. Answers just get slightly worse, and you find out from a customer.

The question I care about is narrower: can you lower the cost and prove that users get the same quality?

On Auto-RAG and MerlinX, the agent platforms I work on, a change of this kind reduced cost per conversation by 73% across 840 agents, with no measurable quality loss on a frozen suite at 95% confidence. Here is the method, step by step. Your figures will differ. The order of the steps is what I would keep.

Step 1: Find out where the money actually goes

Most teams know their monthly total. Few can say what one conversation costs, or which step drives it.

Break the cost down per workflow step: orchestration, retrieval, reranking, tool calls, final answer, and any retries. Log model, input tokens, cached tokens, output tokens and step name for each call.

When I did this for our agents, one finding made the whole project simple: the orchestrator model accounted for 75% of workflow cost. That told us where to look. Without a breakdown like that, it is easy to spend weeks tuning the wrong part.

In multi-step agents the expensive call is often one nobody reads. An orchestrator may run several times per conversation, each time with a large context.

Step 2: List the options in order of risk

Take the free wins first. Then change one thing at a time, or you will not know what caused a result.

Step 3: Freeze the evaluation before you test anything

The word "frozen" is doing real work here. The test cases, scoring and pass criteria must be fixed before you see any results for the cheaper option.

If you adjust the suite after seeing results, even with good intentions, you will drift towards the answer you want. Everyone who wants a lower bill has that bias, and that includes me.

The suite should come from real traffic, cover your difficult cases, and score behaviour rather than wording. My RAG evaluation guide covers how to build one.

Step 4: Ask the right statistical question

A common mistake is to run both models, see similar average scores, and conclude they are equal. That proves little, because small test sets can hide real differences.

Models are stochastic, so run each case several times. Then frame the test around the question you care about: is the cheaper option not worse than the current one, by more than a margin you agreed in advance, at a stated confidence level?

"No significant difference" and "proven not worse" are different claims. The first can simply mean you did not test enough. I also state the confidence level whenever I quote the result, because a figure like 73% means little without it.

Look past the average, too. A cheaper model can match on the mean and fail badly on one customer or question type. Check results by segment before you trust the total.

Step 5: Measure total cost per accepted result

Token price is not your cost. A cheaper model that fails more tool calls, retries more often or escalates to a fallback may cost more per finished task.

The metric I would use is cost per accepted outcome, including retries, fallbacks and any extra calls. Run this on the same suite, not on a price sheet.

Step 6: Roll out with a way back

Passing the suite earns a rollout, not a finished job. Ours used a guarded apply with a one-command rollback. For a change like this I would want:

• A gradual rollout by wave or customer, with higher-risk customers later.
• A soak period where error rates and quality signals are compared against the baseline.
• A rollback a person can run in one step, tested before it is needed.
• A record of exactly which agents changed and when.

The rollback has a second benefit. As the Fable 5 suspension showed, you may need to change models for reasons that have nothing to do with price. The machinery is the same.

A worked example, with invented numbers

Here is how the reasoning plays out. These figures are made up for illustration.

Suppose a conversation costs $0.040 on average, and $0.030 of that is the orchestrator. A cheaper orchestrator costs a fifth as much per call, so the price sheet promises a conversation at $0.016, which is 60% less.

The saving is real, at about 46%, but it is smaller than the 60% the price sheet suggested. The more serious line is the drop in accepted conversations. Thirty extra failures per thousand might be fine for an internal tool, and unacceptable for a customer-facing agent. That is a business decision, and it should be made with this table in front of the people who own the outcome.

This is why I would never present a cost result as one number. Show what it cost, what changed, and what you accepted in exchange.

Step 7: Keep measuring

Costs drift back. Prompts grow, new tools add schema tokens, and cache hit rates fall when someone edits the top of a system prompt. Put cost per conversation on the same dashboard as latency and errors, and alert when it moves.

What I would avoid

• Choosing from a pricing table. Price per token says nothing about your workload.
• Trusting a benchmark rank. Public scores tell you what is worth testing, nothing more.
• Testing on easy cases. Cheap models are fine at what is easy. Your suite exists for the rest.
• Announcing savings before quality is proven. Reversing that decision in public is unpleasant.

The order is simple: measure, take the safe wins, freeze the test, prove the change, and roll out so you can undo it. It takes longer than switching a model name, and it is why the result can be defended when a customer asks.

Written by Eslam Hamed, a Technical Lead and AI Solutions Architect working on enterprise RAG, agentic AI, and model evaluation.