Generating Embeddings at Scale: Cheapest Path for Billions of Vectors
A cost-focused guide to generating embeddings at massive scale, comparing hosted APIs, self-hosted GPU batch jobs, and the hidden costs of storing and serving billions of vectors.
Embedding generation looks deceptively simple. You take text, run it through a model, and get back a vector. The trouble starts when there are billions of text chunks to process. At that scale, small per-unit costs and inefficiencies multiply into large bills, and the cheapest path is rarely the most obvious one. This guide walks through the real cost drivers of embedding at scale and how to choose between hosted APIs and self-hosted GPU batch jobs.
Why Embeddings Are a Throughput Problem
Chat inference is latency-sensitive: a person waits for the answer. Embedding generation is almost always offline and batch oriented. Nobody is waiting for a single vector. That difference changes the optimization target entirely. Instead of minimizing time to first token, you want to maximize tokens processed per dollar. Embedding models are also far smaller than chat models, often a few hundred million parameters rather than tens of billions, which means a single GPU can process them at very high throughput when fed large batches.
Because the work is batchable and the models are small, embedding is one of the few inference workloads where self-hosting on rented GPUs can be dramatically cheaper than per-token API pricing, provided you can keep the hardware busy.
Hosted API Versus Self-Hosted Batch
The decision comes down to volume, engineering capacity, and how steady your workload is.
Hosted Embedding APIs
Hosted APIs charge per token or per million tokens and require no infrastructure. They are the right choice when your volume is modest, bursty, or unpredictable, and when engineering time is more valuable than the marginal cost savings. Many providers offer a dedicated batch tier that processes large jobs asynchronously at a meaningful discount compared with the real-time endpoint. If you go the API route for a big job, always check for a batch or async pricing tier before paying real-time rates.
Self-Hosted GPU Batch Jobs
If you have hundreds of millions or billions of chunks to embed, renting GPUs and running your own batch pipeline often wins on cost. The key is utilization. A GPU rented by the hour only saves money if it is processing near its maximum throughput. The economics work like this:
- Choose an efficient open embedding model sized to your quality needs.
- Rent GPUs on-demand or, better, use interruptible or spot capacity for fault-tolerant batch work.
- Feed large batches so the GPU stays saturated rather than waiting on data.
- Process the entire backlog, then release the hardware so you stop paying.
Spot and interruptible instances are especially well suited to embedding because the work is restartable. If an instance is reclaimed mid-batch, you simply re-queue the unfinished chunks. That fault tolerance unlocks the cheapest GPU pricing tiers without real risk.
The Cost Drivers People Forget
The embedding model is only one line in the budget. Several other costs frequently dominate at scale.
| Cost area | Why it adds up | How to control it |
|---|---|---|
| Compute | GPU hours or per-token API fees | Maximize batch utilization, use spot |
| Vector storage | Billions of vectors consume large memory and disk | Use lower dimensions or quantization |
| Index serving | Keeping a vector index queryable costs ongoing memory | Tiered storage, compressed indexes |
| Re-embedding | Model upgrades force a full re-run | Version vectors, plan migrations |
| Data egress | Moving vectors between clouds incurs fees | Co-locate compute and storage |
Dimensionality Is a Storage Multiplier
Each extra dimension multiplies storage and memory across every vector you keep. A model that outputs higher dimensional vectors may improve retrieval quality slightly, but at a billion vectors that choice can multiply your storage bill. Some modern embedding models support truncating output dimensions with only modest quality loss. Test whether a shorter vector meets your retrieval needs before committing to the full width, because the savings compound across the entire corpus.
Quantization and Compression
Storing vectors in full precision floating point is wasteful for most retrieval tasks. Quantizing vectors to lower precision, or using product quantization inside the index, can shrink storage by a large factor while keeping retrieval quality acceptable for most applications. The right approach depends on your accuracy tolerance, so validate recall on a representative query set before rolling quantization out across billions of records.
Plan for Re-Embedding
One cost that catches teams off guard is re-embedding. When you upgrade to a better embedding model, the new vectors are not comparable with the old ones, so the entire corpus must be regenerated. At billions of vectors that is a major compute event. Two habits reduce the pain. First, store the model version alongside every vector so you always know what produced it. Second, treat embedding model selection as a long-term commitment and avoid switching for marginal quality gains. When you do migrate, run the batch on the cheapest fault-tolerant capacity you can find.
A Practical Decision Path
- Estimate total tokens to embed and how often the corpus changes.
- For modest or bursty volume, use a hosted API with a batch tier.
- For very large one-time or steady jobs, rent GPUs and run a saturated batch pipeline on spot capacity.
- Choose the smallest vector dimension that meets retrieval quality.
- Quantize stored vectors and co-locate compute with storage to avoid egress.
- Version every vector so future re-embedding is predictable.
Chunking Choices Affect the Bill
How you split documents into chunks before embedding has a quiet but real cost impact. Smaller chunks mean more vectors, which raises both the embedding token count and the storage footprint. Larger chunks mean fewer vectors but can dilute retrieval relevance, since a single vector now represents a broader span of text. The right chunk size balances retrieval quality against vector count, and at billions of records even a modest change in average chunk size shifts your storage and compute totals noticeably. Decide chunking with cost in mind, not just retrieval quality, because the two are linked.
Deduplication is another easy win that teams skip. Large corpora often contain near-duplicate passages, boilerplate, and repeated headers or footers. Embedding and storing all of them wastes compute and storage on vectors that add nothing to retrieval. A deduplication pass before embedding can meaningfully shrink the corpus you pay to process and keep, and it often improves retrieval quality by removing noise from the index.
The cheapest path to billions of vectors is rarely a single decision. It is the combination of keeping hardware busy, storing vectors compactly, deduplicating before you embed, and avoiding unnecessary re-embedding. Get those right and the per-vector cost drops to a level where embedding an enormous corpus becomes genuinely affordable.