
Intro: Why LLM Latency vs Throughput Matters
When teams evaluate a large language model, they often ask a simple question: “How fast is it?” The answer depends on what “fast” means.
A model may process thousands of tokens per second across many requests yet feel sluggish in a chat application. Another model may deliver its first token quickly but generate long responses too slowly for high-volume document processing. These are different performance profiles, and comparing them requires a clear understanding of LLM latency vs throughput.
Latency measures how long a user or system waits. Throughput measures how much work a serving system completes over time. Both matter, but they matter differently depending on the workload.
For an interactive assistant, the first visible response is critical. If a user submits a prompt and waits several seconds before seeing anything, the system feels unresponsive—even if the complete answer eventually arrives quickly. This initial delay is measured by time to first token, or TTFT.
Once generation begins, the speed of subsequent tokens also affects the experience. A response that appears word by word at a steady pace feels usable. One that pauses repeatedly can feel broken or stalled.
Consider a restaurant analogy. TTFT is the time before the waiter brings the first dish to the table. Tokens per second is how quickly the rest of the meal arrives. A long delay before the first dish frustrates customers, but a slow stream of dishes can also make the meal uncomfortable.
For batch workloads, the priorities change. If a system summarizes 100,000 documents overnight, users may not care whether each summary begins streaming immediately. They care about total completion time, resource utilization, and cost. In that setting, throughput and goodput are usually more important than TTFT.
This guide compares the most important LLM performance metrics, including:
- Time to first token, which captures initial responsiveness.
- End-to-end latency, which measures the entire request.
- Inter-token latency, which shows gaps during streaming.
- Tokens per second, which measures generation or processing speed.
- Requests per second, which measures request capacity.
- Goodput, which measures useful work that meets defined service targets.
It also explains how production AI monitoring can reveal queueing, prompt-processing, decoding, and infrastructure bottlenecks. The practical goal is not to choose one universal metric. It is to match each metric to the user experience and operating requirements of a specific application.
Background: Understanding LLM Latency vs Throughput
LLM latency is the elapsed time between a request being sent and a specified response event. That event might be the arrival of the first token, the arrival of the final token, or an intermediate token in a streaming response.
Time to first token measures the delay from request submission to the first generated output token. It usually includes network travel, request queueing, prompt processing, and the initial decoding step.
TTFT is especially important for chatbots, coding copilots, voice assistants, and agent interfaces. A shorter TTFT gives users immediate evidence that the system is working.
A useful example is a search engine. Showing an initial page shell quickly reassures the user that the request succeeded, even if the complete results load afterward. Streaming the first LLM token serves a similar psychological and practical purpose.
End-to-end latency measures the complete interval from sending a request to receiving its final token. It includes TTFT plus the time spent generating the remaining output.
Inter-token latency, or ITL, measures the gap between consecutive output tokens. Its average is closely related to time per output token, often called TPOT. However, individual ITL measurements can reveal pauses that an average hides.
For example, an answer might average 40 milliseconds per token but contain several 800-millisecond stalls. The average may look acceptable while the user experiences an awkward stream. Production AI monitoring should therefore track distributions rather than relying on one average number.
LLM throughput measures how much work a system completes within a period. It can refer to tokens processed, requests completed, or requests that satisfy service-level objectives.
Tokens per second must be labeled carefully. Input tokens per second describe prompt processing, often called prefill throughput. Output tokens per second describe generation during decoding.
These phases have different performance characteristics. A large prompt can increase prefill time, while a long response increases decode time. Optimizing one does not always improve the other.
Output tokens per second is often the metric people mean when they say a model “generates quickly.” Yet it should not be confused with user-perceived responsiveness. A system can generate output rapidly after spending too long in a queue.
Requests per second, or RPS, measures completed requests within a time window. Comparisons are meaningful only when requests have similar prompt lengths, output lengths, and generation settings.
Goodput is more practical for production. It measures the rate of requests that complete while meeting all relevant service-level objectives, such as:
- TTFT below a defined threshold.
- Inter-token latency below a defined threshold.
- End-to-end latency within a target.
- Successful completion without errors.
A server can achieve high RPS while delivering poor goodput if many requests violate latency targets. In other words, throughput counts activity, while goodput counts successful, acceptable service.
Trend: How LLM Performance Metrics Are Evolving
Early LLM benchmarks often emphasized average tokens per second. Modern systems need a more complete view because production traffic is variable. Requests arrive in bursts, prompts differ in size, and users may request very different output lengths.
The mean describes overall average behavior, but it can be distorted by outliers. The median represents a typical request. P95 shows the latency experienced by the slowest 5% of requests, while P99 highlights the slowest 1%.
Tail latency matters because a small percentage of slow requests can affect customer satisfaction, agent workflows, and service-level agreements. A team might report a 300-millisecond median TTFT while some users regularly wait five seconds.
A request timeline helps separate bottlenecks:
- Queueing: time waiting for available capacity.
- Prefill: time processing the input prompt.
- Decode: time generating output tokens.
- Network and transfer: time moving data between client and server.
If TTFT rises while tokens per second remains stable, queueing or prefill may be the problem. If TTFT is acceptable but output streams slowly, decoding capacity may be constrained.
Smaller, specialized models can reduce both cost and latency for well-defined tasks. Quantization can lower memory use and improve serving efficiency, although teams must validate quality and hardware compatibility.
A compact model for classification or routing may outperform a larger general-purpose model operationally, even if the larger model scores better on broad benchmarks.
Reducing unnecessary output is one of the most direct forms of LLM throughput optimization. Concise instructions, structured formats, stop conditions, and appropriate maximum-token limits can reduce generation time.
Reducing input tokens can also help, particularly with very long contexts, but output reduction often has a more visible impact on end-to-end latency. A prompt is like luggage at an airport: reducing it may speed check-in, but the time spent traveling still depends heavily on the journey itself.
Batching improves hardware utilization by processing multiple requests together. However, excessive batching can increase TTFT, creating a trade-off between throughput and responsiveness.
Independent model calls can often run in parallel. Speculative decoding uses a faster draft process to predict likely tokens while a larger model verifies them. These techniques can improve tokens per second without changing the user-facing model.
Streaming does not necessarily reduce total computation, but it lowers perceived waiting time. Users can begin reading, reviewing, or interrupting an answer before generation finishes.
For interactive applications, streaming is often more valuable than a small improvement in final response time. It turns an opaque wait into visible progress.
Prompt caching, response caching, and key-value cache reuse can reduce repeated computation. Combining sequential tasks into one structured request can also remove network round trips.
Still, fewer requests are not always better. A large combined request may become harder to retry, validate, or parallelize. Optimization should consider reliability and maintainability as well as raw speed.
Insight: LLM Latency vs Throughput Comparison
TTFT answers: “How quickly does the response begin?” Tokens per second answers: “How quickly does processing or generation continue?”
| Metric | Best for | Main question | Typical risk |
|---|---|---|---|
| TTFT | Chat and copilots | Does response begin quickly? | Hides slow decoding |
| Output TPS | Streaming responses | Does text generate smoothly? | Hides queueing |
| E2E latency | Complete tasks | How long until finished? | Sensitive to output length |
| Goodput | Production services | How much work meets SLOs? | Requires defined targets |
The key takeaway is that no single metric describes complete LLM performance. Teams should use several metrics together.
Choose TTFT when users are waiting interactively, especially for chat, code completion, customer support, and agent interfaces. A fast first token can improve trust and reduce abandonment.
Choose tokens per second when processing large volumes, generating long outputs, or maximizing accelerator utilization. Batch summarization, synthetic data generation, and offline evaluation are common examples.
Interactive workloads generally prioritize TTFT, ITL, and tail latency. Users want a fast start and a smooth stream. A system that maximizes batch throughput by delaying every request may be operationally efficient but commercially disappointing.
Batch workloads prioritize output TPS, input TPS, RPS, cost per token, and total completion time. They can tolerate queueing if the overall job finishes within its deadline.
Start with the experience, not the infrastructure dashboard. Ask whether users need immediate feedback, fast completion, predictable response times, or maximum volume.
A practical dashboard should include:
- Median, P95, and P99 TTFT.
- Average and tail ITL or TPOT.
- End-to-end latency by output length.
- Input and output tokens per second.
- RPS and goodput.
- Error, timeout, and cancellation rates.
- Cost per successful request.
Forecast: The Future of LLM Performance Measurement
As LLM applications become business-critical, benchmarks will give way to production AI monitoring tied to service-level objectives.
Future dashboards will focus less on a single headline speed and more on latency distributions by model, tenant, region, prompt size, and workload type.
Goodput will increasingly include quality and cost. A fast answer that fails evaluation criteria is not truly useful. Teams will measure successful, accurate responses per dollar and per unit of compute.
Serving systems will adjust batching based on traffic conditions. Dynamic model routing may send simple tasks to smaller models and complex tasks to larger ones, balancing quality, latency, and cost.
A single platform may serve chat, batch jobs, retrieval, and agents simultaneously. Future systems will select optimization policies in real time, prioritizing TTFT for interactive traffic and throughput for offline workloads.
Call to Action: Build a Better LLM Monitoring Plan
Set targets for TTFT, ITL, E2E latency, output TPS, error rate, and goodput. Define them separately for each workload rather than applying one target everywhere.
Capture request start time, first-token time, every token arrival, final-token time, input tokens, output tokens, queue duration, and model identifier. Without these fields, diagnosing performance becomes guesswork.
Test realistic traffic patterns, prompt sizes, concurrency levels, and output lengths. Compare median and tail results before declaring an optimization successful.
Conclusion: Choose Metrics That Reflect Real AI Performance
TTFT and other latency metrics describe responsiveness. Tokens per second, RPS, and goodput describe scale and capacity. Both are essential, but neither replaces the other.
The right metric depends on the job. Improve TTFT for interactive experiences, optimize tokens per second for high-volume processing, and use goodput to confirm that performance improvements actually satisfy users and service objectives.
The future of LLM operations will not be defined by the fastest benchmark number. It will be defined by systems that deliver the right quality, speed, reliability, and cost for each workload.