Prompt Caching: Cut Your LLM API Bill by 90%

AI on a Budget. This is part of our running series on cutting AI costs. See the full series →

Everyone wants to argue about which model is cleverest. Meanwhile the biggest saving in AI right now is dead simple and almost nobody bothers with it: stop paying for the same tokens twice. If you send a chunky system prompt, a style guide, a codebase or a fat document with every single request, you are being billed full whack to re-read text the model already saw thirty seconds ago. Prompt caching fixes that, and the discount is not small. It is up to 90%. This one is part of our AI-on-a-budget series, and it might be the single highest-return hour you spend this month.

Table of Contents

What prompt caching actually does

When you call an LLM, the provider has to process every token in your prompt before it writes a word. That processing is most of what you pay for on the input side. Prompt caching lets the provider store the processed version of a repeated chunk (your system prompt, a long reference doc, few-shot examples) and skip the recompute next time it sees the same prefix. You still send the text, but you are charged a fraction to reuse the cached version.

The rule that trips people up: caching works on prefixes. The cached bit has to sit at the front and be byte-for-byte identical. So you put the stable stuff (instructions, schemas, documents) first, and the changing stuff (the user’s actual question) last. Reorder your prompt and you can go from a 90% saving to nothing.

The numbers, provider by provider

Anthropic (Claude): cache reads cost roughly 10% of the base input price. On Claude Sonnet that is about $0.30 per million tokens instead of $3.00. The catch: the initial cache write costs about 25% more than a normal input token, and the default cache only lives for five minutes (there is a one-hour option). So caching pays off when you reuse the same prefix repeatedly inside a short window, which is most agents, chatbots and document Q&A tools.
OpenAI: caching is automatic and needs no code changes. It kicks in on prompts over 1,024 tokens and knocks up to 50% off the cached portion. You do not opt in, you just structure prompts so the reused part sits at the front.

A worked example. Say you run a support bot with a 4,000-token system prompt and knowledge base, and it handles 50,000 messages a month on Claude Sonnet. Without caching, that stable prefix alone costs you roughly 200 million input tokens at $3, so about $600. Cache it and those reads drop to around $0.30 per million, so about $60, plus a trivial one-off write cost. Same behaviour, 90% off the biggest line on the bill.

How to switch it on

Split your prompt. Static first (instructions, tools, docs, examples), dynamic last (the user turn). This alone unlocks most of the win.
Mark the cache point (Anthropic). Add a cache_control breakpoint at the end of your stable block. OpenAI needs nothing, it is automatic.
Keep the prefix stable. A timestamp, a random ID or a reordered sentence at the top will bust the cache. Move volatile values to the end.
Batch your traffic. Because Anthropic’s cache expires in five minutes, sporadic requests miss. Keep a conversation or a job flowing and you keep hitting the cache.

When it is not the answer

Caching only helps when you genuinely repeat context. One-off, wildly different prompts get nothing, because there is no shared prefix to reuse. If your requests are similar but not identical (users asking the same question ten different ways) that is a job for semantic caching instead, which matches on meaning using embeddings and can shave another 40 to 70% off. And if you can wait, the batch API stacks a separate 50% discount for jobs that can finish within 24 hours. Caching, semantic caching and batching are three different levers, and the frugal move is to pull all three where they fit.

The reason so few people bother is that caching is boring. It does not make your model smarter or your demo flashier. It just removes a zero from your invoice, which, when the invoice is real money, is the most interesting thing an engineer can do all week.

Did you know: the idea of caching a computed result to dodge repeating the work is older than the web itself. The term “cache” comes from the French cacher, to hide, and computer scientists borrowed it in the 1960s for the small fast memory that hides slower main memory from the processor.

Sources:

The free stack. One email a week: the AI tools and moves that actually matter, hype filtered out. Subscribe free →