Batch Inference: How Async Processing Slashes Token Costs
Batch inference trades immediacy for a large per-token discount. This guide explains when async processing fits, how providers price it, and how to estimate the savings.
Not every request needs an answer in two seconds. A nightly summarization job, a backlog of documents to classify, or a content pipeline that refreshes once a day can all tolerate a wait of minutes or hours. Batch inference is the pricing and execution model built for exactly those workloads, and it is one of the most reliable ways to lower your spend on large language model tokens without touching model quality. The core idea is simple: you hand the provider a large file of requests, the provider schedules them when capacity is convenient, and you accept a delayed result in exchange for a meaningful discount.
What batch inference actually is
In an online or synchronous setup, you send one request and block until the tokens come back. Your latency budget is tight, and the provider must hold capacity ready to serve you the instant you call. That readiness has a cost, and you pay for it in the per-token rate. Batch inference inverts the deal. You submit many requests at once, often as a single file, and the provider commits to returning all results within a window such as a few hours up to roughly a day. Because the provider can slot your work into idle capacity and pack requests densely, the marginal cost falls, and that saving is passed back as a lower per-token price.
Discounts vary by provider and model, but a common pattern is a reduction of around half off the synchronous rate for the same model. The exact figure changes, so always confirm current pricing on the provider you are evaluating rather than assuming a fixed number.
When batch inference is the right tool
The deciding factor is your tolerance for delay. If a human or a downstream system is waiting in real time, batch is wrong. If the work is offline, batch is often the cheapest correct answer. Typical good fits include the following.
- Bulk classification or tagging of a large document or product catalog.
- Nightly or weekly summarization of logs, tickets, or transcripts.
- Generating embeddings for a corpus before indexing.
- Evaluation and scoring runs over a fixed dataset.
- Backfilling a feature across historical records.
- Synthetic data generation for training or testing.
Poor fits are anything user-facing and interactive: chat, autocomplete, live agents, or anything with a strict response deadline. For those, you want synchronous endpoints, and possibly continuous batching on the server side, which is a different technique covered separately.
How the savings add up
To estimate savings, separate the two variables you control: volume and rate. Batch does not change the number of tokens you process, only the price per token. So the saving is roughly the volume multiplied by the difference between the synchronous and batch rates.
| Scenario | Monthly tokens | Sync est. cost | Batch est. cost | Approx. saving |
|---|---|---|---|---|
| Small pipeline | 50M | Baseline | About half | Roughly 50 percent |
| Mid catalog refresh | 500M | Baseline | About half | Roughly 50 percent |
| Large backfill | 5B | Baseline | About half | Roughly 50 percent |
The table uses relative figures on purpose. Absolute prices move and differ by model and provider, so the durable lesson is that the percentage discount applies uniformly to volume. The larger and more deferrable your workload, the more dollars that percentage represents.
Hidden factors that change the math
A few practical details can shift your real savings. Input and output tokens are usually priced differently, so a job that is output heavy behaves differently from one that is input heavy. Retries on failed rows add tokens you may not have planned for. And if your batch job feeds a downstream store, the cost of that store and any reprocessing should sit in the same budget so you are comparing total pipeline cost, not just the model line item.
Building a reliable batch workflow
Treat a batch job like any other data job. The steps below keep it predictable.
- Assemble requests into the provider's expected file format, one request per line with a stable identifier you can join on later.
- Validate prompts and token counts before submission so you do not pay for malformed rows.
- Submit the file and record the job identifier.
- Poll for completion rather than blocking a process, since the window can be hours.
- Download results, join them back to your source records by identifier, and reconcile any failures.
- Re-submit only the failed subset rather than the whole file.
Idempotency matters. Because a batch can partially fail, your join key and your re-submission logic should make it safe to run a row twice without corrupting downstream data.
Common mistakes to avoid
The first mistake is using batch for latency-sensitive work and then bolting on complex queuing to hide the delay from users. That usually costs more in engineering time than the discount saves. The second is ignoring the completion window in capacity planning: if your pipeline assumes results in ten minutes but the window is several hours, every dependent job inherits that delay. The third is forgetting that the discount applies to a specific model. Switching to a cheaper model on a synchronous endpoint can sometimes beat batch pricing on a larger model, so compare the full options rather than assuming batch always wins.
Combining batch with other cost levers
Batch pricing stacks neatly with the other techniques that lower inference spend, because it changes the rate while they change the work. A few combinations are worth planning for deliberately.
- Prompt trimming: since you pay per token, cutting boilerplate and redundant context from every row multiplies across a million-row file. Tighten prompts before you submit, not after.
- Right-sizing the model: a batch job rarely needs the largest model. Evaluate whether a smaller model at the batch rate meets your quality bar, since that compounds two discounts at once.
- Caching shared context: if every row reuses the same long instructions or reference document, structure the job so that shared portion is not re-billed needlessly where the provider supports it.
- Output capping: set a sensible maximum output length so a few runaway generations do not inflate the bill, since output tokens often carry the higher rate.
A worked example of the savings logic
Suppose a content team classifies a backlog of one million support tickets once a week. Each ticket plus instructions runs a few hundred input tokens and produces a short label of output tokens. On a synchronous endpoint that workload would carry the full per-token rate and would also tie up capacity that could serve live traffic. Moved to batch, the same volume runs overnight at roughly half the rate, the live endpoints stay free for real users, and the weekly job finishes inside its window with room to spare. Nothing about the model or the labels changed; only the execution model and the price did. That is the essence of why batch is such a dependable lever: it is a pure pricing win on work that was never time-sensitive in the first place.
Putting it together
Batch inference is one of the clearest cost levers in the LLM toolbox because it changes price without changing model behavior. If a workload can wait, moving it to async batch processing commonly removes a large share of its token bill, and the savings scale directly with volume. The right approach is to inventory your inference workloads, separate the interactive from the deferrable, and route everything deferrable through a robust batch pipeline with proper identifiers, validation, and partial-failure handling. Do that, and you keep the quality of the same model while paying a fraction of the rate.