Streaming LLM Responses: Time to First Token and Why It Matters
A practical guide to time to first token (TTFT) and inter-token latency, the two metrics that decide how responsive a streaming LLM feels and how to benchmark them across providers.
When a chat assistant starts typing within a fraction of a second, it feels fast even if the full answer takes several seconds to finish. That perception is driven by streaming, and the single most important number behind it is time to first token, usually shortened to TTFT. If you are comparing inference providers on price alone, you are missing half the story. A cheaper endpoint that makes users wait two seconds before anything appears can feel worse than a slightly pricier one that responds almost instantly.
What Time to First Token Actually Measures
Time to first token is the gap between sending a request and receiving the first chunk of generated output. It captures everything that happens before generation can begin: network travel to the provider, request queuing, model loading if the weights are not already resident, and the prefill stage where the model processes your entire prompt. Only after prefill completes can the model emit that first token.
This matters because prefill cost scales with prompt length. A short system prompt and a one line question prefill almost instantly. A long retrieval-augmented prompt stuffed with documents can take noticeably longer, because the model must run attention across every input token before it produces anything. So TTFT is not a fixed property of a provider. It is a function of your prompt, the model size, the hardware, and how busy the endpoint is.
TTFT Versus Inter-Token Latency
Streaming inference has two latency phases, and conflating them leads to bad comparisons.
- Time to first token (TTFT): how long until the first output chunk arrives. Dominated by prefill and queuing.
- Inter-token latency (ITL): the average gap between subsequent tokens once generation is flowing. Dominated by the decode step and how many requests share the GPU.
A user reads a streamed answer at a certain pace, so once inter-token latency is faster than reading speed, making it faster yields diminishing returns. TTFT, by contrast, is felt every single time. Shaving a second off TTFT often improves perceived quality more than shaving the same time off total generation.
Tokens Per Second Can Mislead
Many marketing pages advertise tokens per second as the headline figure. That number usually reflects steady-state decode throughput and says little about how long users wait before the stream starts. Two providers can advertise similar tokens per second while delivering very different TTFT under load. Always look for both numbers, measured at a realistic prompt length.
What Drives TTFT
Several factors push time to first token up or down. Understanding them helps you choose providers and tune your own requests.
| Factor | Effect on TTFT | What you can do |
|---|---|---|
| Prompt length | Longer prompts mean longer prefill | Trim context, cache shared prefixes |
| Model size | Larger models prefill slower | Route easy tasks to smaller models |
| Cold start | Loading weights adds seconds | Use warm or provisioned endpoints |
| Queue depth | Busy endpoints delay prefill | Check capacity and rate limits |
| Network distance | Far regions add round-trip time | Pick a region near your users |
Prompt Caching Changes the Math
Many providers now offer prompt caching, where the prefill computation for a repeated prefix is stored and reused. If your application sends the same long system prompt or the same document set on every call, caching can cut TTFT dramatically because the expensive prefill work is skipped. When comparing providers, ask whether they support prefix caching, how long entries persist, and whether cached input tokens are billed at a reduced rate. For high-volume applications this single feature can shift both latency and cost.
How to Compare Providers Fairly on Streaming Latency
Headline numbers from vendor pages rarely match what you will see in production. To compare fairly, measure TTFT yourself under conditions that match your workload.
- Use a prompt of realistic length, not a trivial one word input.
- Send requests from the region where your users live.
- Measure during representative traffic, including peak hours, not just an idle test.
- Report percentiles, especially the slower tail, because median TTFT hides the bad experiences.
- Separate TTFT from inter-token latency in your results.
Percentiles deserve emphasis. A provider with a great median but a heavy tail at the slower percentiles will frustrate a meaningful share of users. For interactive products, the tail is often what determines whether the experience feels reliable.
When TTFT Matters Most
Not every workload cares about first token speed. A nightly batch job that summarizes thousands of documents cares about total throughput and cost per token, not whether the first token arrived in half a second. But anything a person waits on in real time lives and dies by TTFT: chat assistants, coding copilots, voice agents, and live search. For voice in particular, where the response feeds a speech engine, a slow first token creates an awkward pause that breaks the illusion of conversation.
Reducing Your Own TTFT
Beyond picking a fast provider, there are several things you control that move time to first token. The most powerful is reducing prompt length, since prefill is where most of the TTFT for long prompts is spent. Trim instructions that repeat, remove redundant context, and retrieve only the documents a query actually needs rather than stuffing everything into the prompt. Each token you remove from the input shortens prefill and brings the first token closer.
A second lever is keeping the endpoint warm. If your traffic allows an endpoint to scale to zero, the first request after idle pays a cold start that shows up as a much longer TTFT. For interactive products, a small amount of always-on or provisioned capacity removes that spike. A third lever is region selection: placing the endpoint near your users cuts the network portion of TTFT, which is pure overhead that adds nothing to answer quality. None of these require a different model, only a more deliberate setup.
The practical takeaway is to match the metric to the use case. For interactive products, weight TTFT heavily and confirm it holds up under load and across regions. For batch and offline work, prioritize cost per million tokens and sustained throughput instead. Streaming is what makes large models feel usable, and time to first token is the number that decides whether that feeling is fast or sluggish. Treat it as a first-class metric alongside price when you evaluate any inference provider.