Fifty people ask your support bot how to cancel. Fifty different phrasings, one answer, and you paid the frontier model fifty times to write it. Semantic caching is the fix for that specific stupidity: embed each incoming question, look for a past question that means the same thing, and hand back the stored answer without calling the model at all.
The savings are real. Redis puts its LangCache figure at up to 70% fewer LLM calls on repetitive traffic, and Respan’s production numbers land at 30 to 50% on conversational workloads. The catch is equally real: at the threshold where the cache starts paying for itself, roughly 3 to 7% of hits are the wrong answer served with full confidence. This instalment of the AI on a Budget series is about getting the 50% without eating the 7%.
How it differs from prompt caching
Provider prompt caching, covered earlier in the series, matches an exact prefix. The model still runs; you pay less for the bit it has already read, and a wrong answer is impossible.
Semantic caching skips the model entirely. Three steps: convert the query to a vector with an embedding model, run a nearest-neighbour search against every stored query, and if the cosine similarity of the closest match beats your threshold, return that match’s stored response. Miss, and you call the LLM as normal, then store the new pair.
The embedding step is nearly free. OpenAI’s text-embedding-3-small costs $0.02 per million tokens; a 30-token support question costs about six ten-thousandths of a cent to embed. The lookup adds 50 to 200 ms of latency, which only matters if your hit rate is too low to earn it back.
The threshold is the whole product
Every semantic cache is one number wearing a trench coat. Respan’s measured figures on a customer-support workload:
- 0.99: 1 to 3% hit rate, false positives under 0.1%
- 0.97: 5 to 10% hit rate, about 0.5% false positives
- 0.95: 15 to 25% hit rate, 1 to 3% false positives
- 0.93: 25 to 40% hit rate, 3 to 7% false positives
- 0.90: 35 to 55% hit rate, 7 to 15% false positives
Break-even, where the embedding overhead is comfortably paid back, sits around a 30% hit rate, which means running at 0.93 to 0.95. Redis’s own guidance is looser: start a FAQ cache at 0.88 and drop to 0.84 if paraphrases are not matching. The gap between vendors tells you nobody’s default is calibrated to your data. A bot handling 10,000 queries a day at 30% hits and 3 to 7% false positives is sending 90 to 200 people a day a wrong answer it believes is right.
Where it wins and where it will hurt you
It wins on many-to-one traffic: support FAQs, documentation Q&A, internal analytics assistants where the same team asks “signups last week” in twelve ways and a wrong answer gets caught by a human anyway. Expect 30 to 45% hit rates at sensible thresholds on these.
It loses on anything where small input differences should change the output. “Sort an array” and “sort an array in descending order” score around 0.94 similarity and need different code. Personalised responses, tool calls with side effects, and anything regulated (legal, medical, financial advice) should stay out of the cache. A confidently wrong cached refund policy is a compliance incident, and it will not show up in your error logs because, as far as the system is concerned, nothing failed.
Two ways to build it
GPTCache (open source, from Zilliz) wraps the OpenAI client and does the embedding, vector store and hit/miss logic for you. Fine for low to moderate traffic; the embedding model becomes the bottleneck at scale.
Redis vector search if you already run Redis: an HNSW index with COSINE distance, a KNN 1 query, and you convert the returned distance to similarity with 1 minus score. Redis also sells this as a managed service, LangCache, with per-entry TTLs, LRU/LFU eviction, metadata filters for multi-tenant isolation and an LLM-as-judge check on hits. It was free in public preview and Redis has illustrated a $1.50 per million input tokens figure for the service, with final pricing marked TBD, so budget on the assumption it will not stay free.
The setup that does not embarrass you
- Layer caches: exact-match first (zero risk), semantic second, LLM last.
- Start at 0.97. Lower it only when a sampled false-positive rate says you can. Respan’s tolerances are 2% for ordinary products, 0.5% for regulated ones.
- Sample 1 to 5% of cache hits and grade them against a fresh model answer, weekly. This is the eval loop; without it you are guessing.
- Strip timestamps, session IDs and boilerplate before embedding, or nothing will ever match.
- Partition by user or tenant for anything personalised. Never share cache across accounts.
- Set TTLs: 15 to 30 minutes for volatile data, days to weeks for stable FAQ content.
- Re-tune the threshold after any embedding model change. Swapping 3-small for 3-large shifts the whole similarity distribution.
- Fail open. If the vector store is down, call the LLM, do not return errors.
Do that and semantic caching is a 30 to 50% cut on the most repetitive slice of your bill. Skip the eval loop and it is a slow leak of user trust that your dashboards will report as a cost win.
Did you know: the cosine similarity behind all of this is the same maths Gerard Salton used for keyword search in the 1970s; the vectors got longer, the formula did not change.
Sources
- Respan: Semantic Cache for LLMs, when to use it, when to skip it (threshold and false-positive data)
- Redis: 10 techniques to optimise your semantic cache
- Redis Cloud docs: LangCache
- OpenAI embeddings pricing 2026
Related on Top Tool Stack: AI on a Budget: cut your AI costs · Prompt caching