The Problem Nobody Talks About

Everyone’s excited about what LLMs can do. Nobody’s talking about the bill.

A single GPT-4o call costs $2.50 per million input tokens and $10.00 per million output tokens. Scale that to 600 documents a day, or 8 million output tokens a month, and you’re looking at six figures annually before you’ve built anything real.

The companies that deployed early are now in their “oh shit” phase. Finance flagged the line item. Engineering got told to fix it. And a pattern emerged: token efficiency is the new DevOps — invisible when it works, catastrophic when it doesn’t.

The Numbers Are Real

Here’s what happened when actual engineering teams decided to optimize:

Company Before After Savings Time
Coinbase — — ~50% Ongoing
B2B SaaS ($2.1M ARR) $48,000/mo $19,400/mo 60% ($344K/yr) 6 weeks
LendFlow (fintech) $15,000/mo $4,500/mo 70% ~4 weeks
Preply — — 46% Rolling
Document processing agent $4,200/mo $1,150/mo 73% 3 weeks
Content generation pipeline $108,000/mo ~$38,000/mo 65% 1 weekend

These aren’t hypotheticals. These are published post-mortems from 2025-2026.

The Five Levers

Every successful optimization I’ve seen uses some combination of these:

1. Model Routing (30-50% savings)

Not every task needs GPT-4o or Claude Opus. Classification, extraction, and formatting tasks perform just as well on GPT-4o-mini (15-20x cheaper) or DeepSeek V4 Flash (9x cheaper).

The pattern: build a lightweight classifier that routes each request to the cheapest adequate model. The classifier itself runs on a cheap model — the cost of classification is negligible compared to the savings on inference.

Real result: LendFlow’s classification step went from $2,880/month to $230/month. 92% of tickets routed to mini, 8% escalated to flagship. Accuracy stayed within 0.3% of the all-flagship baseline.

2. Semantic Caching (20-40% savings)

Most production workloads have repeated patterns. Same system prompt. Same document types. Same query shapes.

Semantic caching stores previous responses and returns them when a new request is similar enough (cosine similarity threshold ~0.92). Cache hit rates of 40-71% are common after warm-up.

Real result: One team’s sidebar widget went from $3,400/month to $980/month — a single change. Hit rate settled at 71%.

3. Prompt Pruning (20-57% reduction)

System prompts accumulate cruft. Historical examples, polite instructions (“please respond accurately”), scaffolding from earlier versions. One team rewrote their four biggest prompts and dropped average input tokens from 1,840 to 720 — same outputs, no quality loss.

Real result: LendFlow split their extraction prompt into a base (2,400 tokens) and document-type-specific appendices. Average prompt size dropped from 4,200 to 2,700 tokens — 36% reduction, directly translated to cost savings.

4. Batch Processing (50% discount)

OpenAI and Anthropic both offer batch APIs at ~50% discount with a 24-hour SLA. For backfills, nightly digests, ETL summarization, and eval runs — anything that doesn’t need real-time responses — this is free money.

Real result: One team migrated their nightly job to batch and immediately halved that workload’s cost. Bonus: it eliminated the 429 rate-limit spikes in user-facing endpoints.

5. Prompt Caching (50-90% off prefix tokens)

Both OpenAI and Anthropic automatically cache repeated prompt prefixes. Cached input tokens cost 50% less on OpenAI and 90% less on Anthropic. This activates automatically — no code changes needed.

Real result: Most teams see a 30-50% reduction in input token costs within the first two days of enabling prompt caching.

The Stack

The teams that saw 60-70% reductions combined multiple levers. Here’s the sequenced playbook:

Week Change Cumulative Savings
1 Prompt caching (free, no-code) 30-50% off input
2 Model routing + semantic caching 50-60% total
3-4 Prompt pruning + format optimization 60-65% total
5-6 Batch migration + per-feature caps 65-73% total

Why This Matters for Agents

LLMs have no cost awareness. They’ll use the same tokens to answer “what’s 2+2” as they will to draft a legal contract. Humans have no cost intuition either — we can’t feel the $0.80 per document burning.

The insight: cost optimization must be architectural, not behavioral.

The teams that succeeded didn’t tell engineers “use fewer tokens.” They built routing layers. Caching layers. Budget caps. The system optimized itself.

The Conclusion

Two quotes from the case studies stuck with me:

“The bill you are staring at is not a tax on using AI. It is a tax on not yet having built the optimization layer.”

“It took two engineers one weekend to save $344K a year.”

Token efficiency isn’t clever prompting. It’s infrastructure. And like all infrastructure, it compounds.

The teams that build it early will scale cheaper and faster than the teams that treat it as an afterthought. DevOps took ten years to become standard. Token efficiency will take two.

Inspired by case studies from Coinbase, Preply, Boundev, AIgateway, and Cloud Cost Cutter. All numbers are published and real.