RAG on a Budget: Cheap Embeddings and Vector Storage

Most RAG bills do not come from the LLM. They come from somebody choosing a managed vector database at the prototype stage, leaving it running, and discovering six months later that a folder of 40,000 PDFs is costing more per month than the humans who wrote them. This is part of the “AI on a Budget” series, and retrieval is where the waste hides in plain sight.

The model you pay to read the retrieved chunks is the expensive bit, so the whole game is retrieving fewer, better chunks and storing them somewhere that costs close to nothing. Embeddings, the step that turns text into searchable numbers, are almost comically cheap now. The database and the dimensions you pick are where you can still burn money.

Table of Contents

Embeddings cost pocket change

OpenAI’s text-embedding-3-small is listed at $0.02 per million tokens on its model page, which is $0.00002 per 1,000 tokens, a fivefold cut from the old ada-002 at $0.0001. The larger sibling, text-embedding-3-large, is $0.00013 per 1,000 tokens, or $0.13 per million. On OpenAI’s own published benchmarks, small scores 62.3 on MTEB against 61.0 for ada-002, and 44.0 on the MIRACL multilingual test against 31.4. So the cheap model beat the old flagship while costing a fifth as much.

Google’s Gemini Embedding costs more, at $0.15 per million input tokens, with a free tier through the Gemini API and AI Studio. It accepts 2,048 tokens per input and held a top spot on the MTEB multilingual leaderboard from March 2025, per Google’s launch post. The free tier is the interesting part for a side project, but free tiers carry rate limits and, on some products, data-use terms, so read them before you pipe client documents through.

Work it out for a real corpus. One million chunks of roughly 500 tokens is 500 million tokens. That embeds for about $10 on text-embedding-3-small, $65 on text-embedding-3-large, and $75 on Gemini Embedding. You will pay that once, plus whatever you add later. The embeddings model page lists the Batch endpoint as supported, and batch pricing across the major providers sits around half price for jobs that can wait, which suits a one-off backfill of an archive. Check the current embeddings rate before you assume the discount applies.

Dimensions are the hidden bill

Storage, not embedding, is what scales your costs. A 1,536-dimension float32 vector takes 6,144 bytes, so a million of them is about 6.1 GB before indexes. Gemini’s default 3,072 dimensions doubles that to roughly 12.3 GB.

Both providers let you truncate. OpenAI’s launch post says a text-embedding-3-large vector shortened to 256 dimensions still beat an unshortened ada-002 at 1,536 on MTEB. Google supports Matryoshka-style truncation and recommends 1,536 or 768. At 256 float32 dimensions, a million vectors weigh about 1 GB. Quality drops as you cut, so test on your own queries, but start by halving the dimensions rather than doubling your database.

Set the dimensions parameter on OpenAI embeddings and benchmark 256, 512 and 1,024 against your own questions.
Store as halfvec in pgvector (2 bytes per dimension instead of 4) to halve disk and index size again.
Keep chunks to 200 to 500 tokens with a little overlap. Bigger chunks make every retrieval call carry more filler to your expensive model.

The database you already own

pgvector is a free Postgres extension. Its README gives HNSW defaults of m = 16, ef_construction = 64 and ef_search = 40, supports halfvec up to 4,000 dimensions, and caps HNSW indexes on the standard vector type at 2,000 dimensions. That last limit bites: Gemini’s 3,072 default will not index as a plain vector, so truncate or use halfvec. One comparison guide puts the practical ceiling at roughly 5 to 10 million vectors before performance degrades, and warns that vector queries compete with your transactional workload on a shared instance.

Pinecone’s free Starter plan gives you 2 GB of storage, 2M write units and 1M read units a month, five indexes, AWS us-east-1 only, plus 5M tokens a month of its hosted embedding models. The catch is the step up: the Standard plan has a $50 per month minimum. If you outgrow 2 GB, you are paying fifty dollars before your first extra query, and that is the cliff to plan around. Qdrant and Weaviate are free to self-host in a Docker container, at the price of you running it.

The honest ranking for most small teams: pgvector in the Postgres you already pay for, Pinecone Starter if you want zero ops and fit under 2 GB, self-hosted Qdrant if you need heavy metadata filtering. Do not start with a dedicated managed cluster for a corpus under a million chunks.

Spend the savings on retrieval quality

Cheap embeddings let you do what the expensive stack makes painful: re-embed when you change chunking. Send only the top 3 to 5 chunks to the model, not the top 20. Pair this with prompt caching on the stable instructions, and the retrieved context becomes the only variable you pay full price for. Add a relevance score floor so a weak match returns “I don’t know” instead of a confident invention.

What this means

Embedding a million chunks costs $10 to $75 once. Do not optimise this line.
Truncate dimensions and use halfvec. Storage is where a RAG bill grows.
Start on pgvector or a free tier. The next real jump, Pinecone’s $50 minimum, should be a decision, not a surprise.
Retrieve fewer chunks. Every extra chunk is billed at your LLM’s input rate on every question.

Do this on a small budget in an afternoon: embed your docs with text-embedding-3-small at 512 dimensions, store them in pgvector as halfvec, retrieve the top 4 chunks. Expect an embedding bill under $15 for a million chunks and a storage footprint under 1.5 GB.

Did you know: OpenAI’s text-embedding-3-small costs $0.00002 per 1,000 tokens, so embedding the entire text of a 100,000-word novel would cost well under half a penny.

Sources

Related on Top Tool Stack: AI on a Budget: cut your AI costs · Prompt caching

The free stack. One email a week: the AI tools and moves that actually matter, hype filtered out. Subscribe free →