Here’s the paradox sitting in your finance review: token prices fell roughly 280x over two years, yet total enterprise AI spend rose 320%, and the median enterprise monthly LLM bill grew 7.2x year over year heading into Q1 2026. Cheaper tokens did not buy cheaper bills. They bought more consumption.
So if your LLM bill doubled this quarter, you don’t have a pricing problem. You have a usage-pattern problem, and it responds to the same discipline cloud bills did a decade ago. The headline number worth knowing before anything else: semantic caching plus model routing cuts API call volume 30% to 50%, no architecture rewrite required.
We’re gmware, a software development firm headquartered in Austin, TX with engineering centers in Bangalore and Mohali, India. We’ve run cloud FinOps engagements for years, and LLM cost optimization is the same job with new units. This is the playbook: why bills explode, where the money goes, the nine levers, and the audit we run first.
The token-price paradox in four numbers
Your bill rises while token prices fall
Call it the token-price paradox: per-token prices collapse, total spend climbs anyway. The ~280x price drop against 320% spend growth happens because cheaper inference makes new usage patterns economical, and those patterns are token-hungry.
The biggest culprit is agents. A chatbot answered once per user message. An agent plans, calls tools, checks its own work, and retries. Watch one in production telemetry and a single user request fans out into ten or twenty metered calls before anyone sees an answer. We see that fan-out in our own agent builds weekly, and the category is only growing: the AI agents market hits $10.9B to $12B in 2026, up from $7.6B in 2025 at a 44% to 46% growth rate.
Multiply per-task fan-out by always-on features and you get a bill that grows faster than usage looks like it should.
Where the money actually goes
LLM spend decomposes into four buckets in nearly every audit we run: realtime chat or assistant inference, agent workflow fan-out, RAG context tokens, and the supporting infrastructure around them. The infrastructure is the small one. Vector database hosting runs $100 to $2K a month. Context is the sneaky one. Every RAG request ships retrieved chunks along with the question, so a sloppy retrieval setup quietly triples input tokens on every single call. (We covered the build-side economics in our RAG implementation cost guide.)
The scale is mainstream now: 73% of enterprises spend more than $50K a year on LLMs, and 37% spend over $250K. Even one production chatbot runs $400 to $6,000 a month in operating costs, and our chatbot cost breakdown itemizes that. Most companies aren’t running one. They’re running a portfolio nobody owns.
LLM spend is a real budget line
Monthly operating cost, per workload
There’s a fifth bucket nobody budgets: retries, evals, and non-production traffic. Failed calls get retried, evaluation suites re-run on every deploy, and dev environments hammer the same APIs as production. Tag them separately or they hide inside “product usage” forever.
The nine levers of LLM cost optimization
Nine levers cover practically every optimization we’ve shipped. Only the first two carry a sourced market number; the rest we’ve marked honestly by where the savings come from.
| # | Lever | What it does | Expected effect |
|---|---|---|---|
| 1 | Semantic caching | Serves repeated and near-duplicate queries from cache, not the API | With routing: 30% to 50% fewer API calls |
| 2 | Model routing | Sends easy queries to cheap models, escalates hard ones | Counted with caching above; up to ~73% cost cut in high-repetition workloads |
| 3 | Prompt diet | Trims bloated system prompts, few-shot examples, boilerplate | Proportional to your input-token share; measure first |
| 4 | Right-sized models | Small models for narrow tasks: classify, extract, route | Large per-call price gaps between model sizes; quality-test per task |
| 5 | Batching | Moves non-realtime work to provider batch endpoints | Batch pricing discounts; varies by vendor |
| 6 | Retrieval hygiene | Caps top-k, dedupes chunks, trims context windows | Attacks RAG context bloat at the source |
| 7 | Output controls | Max-token caps, structured outputs, stop sequences | Cuts output tokens and retry loops |
| 8 | Token budgets + showback | Per-team, per-workload quotas with visible spend | Stops regression; makes waste an owner’s problem |
| 9 | Loop guards + anomaly alerts | Caps agent iterations, alerts on spend spikes | Catches runaway agents before the invoice does |
Which levers to pull first
Sequence matters more than the lever list. First, instrument: separate API keys per workload so the bill decomposes, because you can’t fix a number you can’t attribute. Second, caching and routing, because the 30% to 50% call-volume reduction needs no product changes and the payoff is immediate; in high-repetition workloads like support, reductions have reached about 73%. Third, prompt diet, because engineers stuff system prompts during development and nobody ever deletes anything. Fourth, right-size models per task.
What we’d push down the list: fine-tuning a smaller model to replace a big one (real savings, long path, do it after the cheap wins) and switching providers for a marginal rate difference. Provider migrations burn engineering weeks to save percentages that lever one beats for free. The cheap levers fund the expensive ones.
Pull the levers in this order
A before-and-after, modeled
Here’s a modeled example, not a quote, and your mix will differ. Assume a mid-size product spending $20K a month, applying caching and routing at the bottom of the sourced 30% to 50% band to chat traffic, a 20% prompt trim (measure yours), retrieval caps halving RAG context tokens, and loop guards trimming agent waste:
| Line item | Before | Lever applied | After (modeled) |
|---|---|---|---|
| Realtime chat inference | $9,000 | Semantic cache + router, 30% fewer calls | $6,300 |
| Agent workflows | $5,000 | Loop guards + output caps | $4,000 |
| RAG context tokens | $4,000 | Retrieval caps + chunk dedup | $2,000 |
| Vector DB + supporting infra | $2,000 | Right-sized index and tiers | $1,500 |
| Total | $20,000 | applied | $13,800 |
Modeled monthly bill: before vs after
That’s roughly 30% off without touching product behavior, and it’s deliberately conservative. We used the bottom of the caching band and modest assumptions everywhere else. The point isn’t the exact figure. It’s that the reduction comes from four small, boring changes, none of which your users will ever notice.
Keeping the bill down once it’s down
Governance is what separates a one-off cleanup from a cost structure. The cloud FinOps playbook transfers almost verbatim: token budgets per workload, showback so the owning team sees its own spend, anomaly alerts that page someone when a workload doubles overnight, and a standing rule that every new AI feature ships with a cost model and an owner. We hold cloud migrations to the same discipline, and our cloud migration cost guide makes the same argument in different units.
Do this now rather than later, because the pressure only builds: 72% of enterprises expect their LLM spend to keep rising. Optimization without governance just resets the clock on the next explosion. Governance without optimization locks in today’s waste. You want both, and they’re cheaper together: the instrumentation that finds waste is the same instrumentation that prevents it.
How AI FinOps differs from cloud FinOps
The mechanics transfer, but three things change. First, the unit: tokens, not instance-hours. Token consumption is set by prompt design and model behavior, which means engineers control the bill in ways finance can’t see from the invoice. Second, volatility: a model upgrade or a prompt edit can move consumption sharply overnight, in either direction. That never happened to a reserved instance.
Third, and most important, the quality coupling. In cloud FinOps, a smaller instance is just slower. In AI FinOps, a smaller model can be wrong. Every cost cut needs an eval gate proving quality held, which is why levers 1 through 7 in the table all quietly assume an evaluation harness exists. Cut costs without evals and you’ll find out from your customers.
How gmware runs an AI cost audit
Our audit starts with a blunt question. What are your five most wasteful workloads? Week one is instrumentation and attribution, tagging spend by workload, agent, and feature until the bill decomposes. Week two ranks the waste and ships the quick wins: caching, routing, prompt diet, loop guards on anything agentic. Structural work like model right-sizing, retrieval redesign, and batch migration gets a costed backlog with projected savings per item, so your CFO can see payback before approving the work. Two patterns we keep finding, for what it’s worth: dev and staging traffic billing against production keys, and a forgotten cron job re-summarizing the same unchanged documents every night. Boring waste is still waste. It’s also the easiest kind to delete.
The delivery math mirrors the rest of our practice: senior engineers in Bangalore and Mohali through our AI agents and LLM integration team, oversight from Austin, and the FinOps habits we built doing cloud cost work for years before tokens were the unit.
If your AI bill grew faster than your AI usage this year, get in touch and we’ll tell you within 48 hours whether an audit will pay for itself. We’ll say so plainly if it won’t.