Every big provider sells the same discount and almost nobody claims it. OpenAI, Anthropic and Google will each take exactly 50% off your token bill, input and output, on any request you are willing to wait for. No negotiation, no enterprise contract, no minimum spend. You put your prompts in a file, hand the file over, and collect the answers inside 24 hours at half price.
It goes unclaimed because most teams build every pipeline as though it were a chat window, since that is how the tutorials are written. Your nightly classification job does not need a first token in 400 milliseconds. Neither does your eval harness, your document summariser, or the enrichment script grinding through 80,000 CRM rows on a Sunday morning. Those are batch jobs wearing a synchronous costume, and you are paying double for the costume. This piece is part of our AI on a Budget series.
The offer, provider by provider
OpenAI. Upload a JSONL file where each line is one request, POST it to /v1/batches with completion_window: "24h", then poll until it finishes. Fifty per cent off in both directions. A single batch takes up to 50,000 requests and the input file can run to 200 MB. You can create 2,000 batches an hour. Batch rate limits sit in a separate pool, so a heavy overnight job does not eat the per-model limits your live traffic depends on. If a batch cannot finish inside the window it comes back marked expired and you are billed only for the rows that completed.
Anthropic. The Message Batches API takes up to 100,000 requests or 256 MB per batch, whichever you hit first, at the same flat 50% off. Results stay retrievable for 29 days, which matters more than it sounds when your collection script falls over at 3am and nobody notices until Thursday.
Google. Gemini Batch Mode covers all paid models with the same 50% cut and higher rate limits than the synchronous path. Google bills only requests that complete token processing, so rows killed by safety filters or malformed input never land on the invoice.
Where it gets properly cheap
Batch discounts stack with prompt caching, and that is where the arithmetic turns silly. If ten thousand batch entries share the same 20,000-token system prompt, you pay the cache write once and every subsequent row reads from cache, at half the already-reduced cache read rate. Anthropic applies the 50% batch discount to cache reads and writes as well, which drives stacked input costs into the region of 95% below list on the cheaper models. We covered the caching half of this in the prompt caching instalment. Batch is the multiplier on top.
OpenAI has a middle option worth knowing about. Set service_tier: "flex" on an ordinary synchronous call and you get batch pricing with slower, variable latency and the occasional resource-unavailable error, no file wrangling required. On o3 that means $5 and $20 per million tokens instead of $10 and $40. It is still in beta with limited model coverage, so treat it as a convenience rather than a plan.
The bill you pay instead
Batch is more work, and anyone telling you otherwise has not run one in production. You lose streaming, so debugging a bad prompt means waiting hours to see the damage spread across 50,000 rows. You need line-level error handling, because individual requests fail independently and you get back a results file with holes in it. You need idempotency and a retry path for expired batches. And you need somewhere to hold state between submission and collection, which in practice means a small queue and a cron job you now own forever.
Rough rule: if the job is a few thousand requests and runs once, the engineering time costs more than the tokens you save. Above roughly 10,000 requests a day, the maths flips hard in the other direction.
An afternoon of work
- List every LLM call in your stack and tag each one live or deferrable. Be honest with yourself: “the dashboard refreshes at 9am” is deferrable.
- Take the largest deferrable job and rewrite it as JSONL, one request per line, with a stable
custom_idon every row so you can rejoin results to your source records. - Sort the rows so identical prefixes cluster together, which gives the cache its best chance of hitting.
- Submit, poll on a timer rather than a tight loop, and write the raw results to storage before you parse anything.
- Log which
custom_idvalues came back empty and requeue them in the next run rather than retrying inside the same window.
Skip it if
Anything a human is waiting on stays synchronous. Anything where the next input depends on the previous output stays synchronous too, because a batch has no conversation. And if your monthly API spend is under about $200, ignore all of this and go pull the caching lever instead, which takes an hour and needs no new infrastructure.
Did you know: OpenAI’s Batch API carries no output token limit at all, which is why high-volume long-form generation often runs cheaper and faster through the batch queue than through the front door.
Sources
- OpenAI, Batch API guide
- OpenAI, Flex processing
- Google Developers Blog, Batch Mode in the Gemini API
- Finout, Anthropic API pricing 2026: models, caching, batch
Related on Top Tool Stack: AI on a Budget: cut your AI costs · Prompt caching