Quantization stores a model's weights, the learned numbers inside it, with less detail. Each number then takes fewer bits, the basic units of computer storage. The model needs less memory and often runs faster on the same hardware. Lower costs per answer become possible, but only if the answers still meet your standard.
What quantization means
Think of rounding a long decimal to a shorter one. The value is less exact, but it still does nearly the same job. Quantization applies the same idea to every weight in a model.
The model keeps the same structure and works in the same way. You only change how finely each number is stored. Many common methods skip full retraining, though some need a small set of example inputs to pick good rounding.
Why memory and speed can improve
Memory comes first. A model must fit in the memory of the device running it, often the video memory on a graphics processor, or GPU, which is called VRAM. Smaller weights take less space, so a model that needed a large card may fit on a smaller one.
Speed often follows. Generating text means producing it a token at a time, where a token is a small piece of text, such as part of a word. Each step reads many weights, and moving that data is often the slow part. Fewer bits per weight means less data to move, so the same hardware can often produce more tokens per second.
Gains depend on the hardware and the software running the model. Some setups gain little, so measure instead of assuming. For choosing the hardware itself, the guide to choosing a GPU covers memory sizes.
The quality trade-off
Lower precision rounds away some detail, so the model's answers can shift. Small shifts may go unnoticed in everyday use. Larger shifts show up as more errors, awkward phrasing, or weaker reasoning on hard tasks.
How much quality you lose depends on the model, the method, and the task. A model that writes casual summaries may hold up well, while a model that writes code or follows strict output formats may not.
The more aggressive the reduction, the greater the risk. Pushing to the lowest precision a method offers is rarely the right first choice.
Common approaches in words
Several approaches exist, and they differ in effort and in the quality they keep. Names vary between tools, so read what your provider actually offers.
- Post-training rounding, the usual starting point, which needs no new training run.
- Calibrated methods, which use a small set of example inputs to choose better rounding and often keep more quality at the same size.
- Quantization-aware training, where the model learns with low precision from the start. It can give better results but takes far more effort.
- Mixed precision, which keeps the most sensitive layers, the stages of the network, at higher precision and reduces the rest.
How to test quality before switching
A short, fixed test gives you more than a general impression. Run these steps before you move any real traffic.
- Collect real prompts that matter for your work, such as a support question or a code request.
- Run the original and the quantized model on the same prompts, with the same settings.
- Compare the answers side by side against a simple rubric for accuracy, format, and tone.
- Measure speed and memory on your own hardware, at the load you expect.
- Keep a set of hard cases that failed before, and retest them after every change.
Move a small share of traffic first, and watch errors before going further.
Limits and common mistakes
Quantization does not fix a model that is wrong for the job. It only changes how much room the model needs.
- Judging quality from a single prompt.
- Assuming the same speed gain on every machine.
- Forgetting that long inputs need extra memory for the context, which grows apart from the weights.
- Expecting speed gains when the software has no support for the chosen format.
- Ignoring that some tasks, such as strict formats, are more sensitive than others.
- Switching all traffic at once instead of a small share first.
Support for formats differs across tools and hardware, so treat any general claim with care.
What to check next
Start by confirming which precision formats your provider or hosting setup supports. If you run your own hardware, compare options on the GPU providers board.
If you use a hosted model instead, look at the LLM API providers board. Pricing and availability differ by provider and by region, so check each provider's own terms before you decide.