
Intro: Why LLM Inference Benchmarking Needs More Than Throughput
LLM inference benchmarking is often reduced to a single question: how many tokens can a system generate per second? That number is useful, but it is not enough to determine whether a model-serving system is fast, affordable, or dependable in production.
A platform can achieve impressive token throughput while users still wait several seconds before seeing the first response token. Another system may deliver excellent latency at low concurrency but become unstable during traffic spikes. A highly utilized GPU may also produce poor user-facing performance if requests are queued too long or if long prompts consume memory bandwidth inefficiently.
The practical objective is therefore not to find the fastest system in isolation. It is to understand the relationship between cost, latency, throughput, quality, and reliability under a workload that resembles real usage.
Token throughput measures the volume of text generated over time. It is valuable for capacity planning, batch processing, and comparing infrastructure efficiency. However, it can hide several important details:
- Whether the measurement refers to input tokens, output tokens, or both
- Whether requests were processed concurrently or sequentially
- How long users waited for the first token
- How many requests violated latency targets
- Whether the model quality remained consistent
- Whether the result depended on unusually large batches
Consider two systems. System A generates 1,000 output tokens per second but takes 2.5 seconds to produce the first token. System B generates 700 tokens per second but begins responding in 400 milliseconds. For an interactive assistant, System B may feel considerably faster even though its aggregate throughput is lower.
This is similar to comparing a highway by the number of cars it moves per hour without measuring how long an individual driver waits at the entrance. Capacity and experience are related, but they are not identical.
A useful LLM inference benchmark should measure at least three dimensions:
- Cost: How much infrastructure is consumed per request, token, or successful task?
- Latency: How quickly does the system begin and complete a response?
- Reliability: How consistently does it meet defined service-level objectives?
The analysis below covers AI inference cost analysis, model performance testing, GPU utilization, and LLM reliability metrics. It also explains why production benchmarks must include workload variation, tail latency, quality targets, and failure behavior.
The central principle is simple: benchmark the trade-offs, not just the peak number.
Background: What Is LLM Inference Benchmarking?
LLM inference benchmarking is the structured measurement of how a language model and its serving stack respond to defined workloads. The serving stack can include the model, tokenizer, inference engine, GPU or accelerator, networking layer, batching scheduler, and application gateway.
A benchmark normally specifies:
- The model and model version
- Prompt and completion lengths
- Request arrival pattern
- Concurrency or target request rate
- Hardware and software configuration
- Sampling settings
- Quality requirements
- Latency and error targets
- Measurement methodology
The result should be reproducible enough to support a meaningful comparison. Changing the model, quantization format, batch policy, or prompt distribution can change the outcome substantially.
Inputs should reflect expected production traffic. A customer-support assistant may use short questions and medium-length answers, while a document-analysis system may process long prompts and generate concise summaries.
Outputs also need a target definition. A benchmark that allows one system to produce shorter answers than another may make the shorter system appear cheaper and faster. Quality targets help control this problem. They can include accuracy thresholds, structured-output validity, task completion rates, or human-evaluation criteria.
A benchmark without a quality target is like comparing two delivery services while allowing one to deliver only half the package. The speed result may be accurate, but the comparison is not useful.
Workload design should include realistic prompt lengths, output lengths, concurrency levels, and arrival patterns. Hardware details should include accelerator type, memory capacity, number of devices, interconnect, host CPU, and memory.
Software variables matter as well. Record the inference engine, driver versions, kernel optimizations, quantization method, parallelism strategy, batching settings, and maximum sequence length.
A fair model performance test changes one major variable at a time wherever possible. Otherwise, it becomes difficult to determine whether an improvement came from the GPU, the model, the serving engine, or a configuration change.
An LLM request has several measurable stages. Treating the entire request as one duration makes it harder to identify bottlenecks.
Before the first output token arrives, a request may experience network travel, admission delays, queueing, and prompt processing. Prompt processing is commonly called prefill. During prefill, the model reads the input context and prepares the internal state required for generation.
Time to first token, or TTFT, measures the interval from request submission to receipt of the first output token:
TTFT = t_first - t_send
TTFT is especially important in interactive applications. It determines how long a user sees a blank screen before the system demonstrates that it is working.
Long prompts often increase prefill time. High concurrency can increase queueing. Large batches may improve overall GPU efficiency but delay individual requests. These effects should be separated in the benchmark rather than hidden inside one average latency number.
After prefill, the system enters the decode phase and generates subsequent tokens, usually one step at a time. The delay between consecutive tokens is called inter-token latency, or ITL.
Time per output token, or TPOT, summarizes the streaming portion of the response. For a request with N_out tokens, a common formulation is:
TPOT = (E2E - TTFT) / (N_out - 1)
End-to-end latency measures the complete request:
E2E = t_last - t_send
A response can have excellent TTFT but poor TPOT, causing the user to wait between words or sentences. It can also have slower TTFT but fast streaming afterward. Both patterns may produce the same end-to-end latency while creating different user experiences.
Average latency is easy to understand but can conceal serious outliers. A practical benchmark should report:
- Mean: Overall arithmetic average, sensitive to extreme values
- Median: Typical request behavior
- P95: Latency that 95% of requests meet or beat
- P99: Tail behavior affecting the slowest 1% of requests
These statistics should be calculated separately for TTFT, TPOT or ITL, and end-to-end latency. A system may have an acceptable median but an unacceptable P99 because of queue contention, memory pressure, or scheduling behavior.
For a high-volume service, the tail matters. If a system serves one million requests per day, even a small percentage of slow requests affects thousands of users.
Goodput measures the rate of requests that complete while meeting all defined requirements. Those requirements may include TTFT, end-to-end latency, output validity, and error limits.
Goodput = N_good / (t_end - t_start)
Goodput cannot exceed raw requests per second. The difference represents requests that completed too slowly, failed, or violated another target.
This metric is often more meaningful than throughput. A platform that completes 100 requests per second but satisfies service objectives for only 70 of them has an effective goodput of 70 requests per second.
Trend: Why Inference Evaluation Is Moving Beyond Speed
Production systems increasingly optimize for user-perceived responsiveness rather than maximum laboratory throughput. Streaming interfaces make TTFT and ITL visible. Users notice the initial blank-screen delay and pauses between tokens even when the final answer arrives quickly.
This shift is particularly important for agents. An agent may make several model calls during one task, so a small latency penalty can multiply across tool selection, planning, retrieval, and response generation.
Standardized benchmarks improve comparability by defining datasets, quality targets, request patterns, and reporting rules. However, a single standardized scenario cannot represent every production application.
A useful evaluation combines common reference tests with workload-specific scenarios.
Closed comparisons hold core variables constant to support “apples-to-apples” analysis. They are useful when comparing hardware or serving frameworks using the same model and quality target.
Open comparisons allow different models, retraining methods, or optimization techniques. They encourage innovation but require more careful interpretation. A faster result may reflect a smaller or differently trained model rather than a superior serving system.
A benchmark dataset should represent the domain, language mix, prompt lengths, and safety requirements of the intended application. Quality validation is essential because aggressive quantization or speculative decoding may improve speed while changing output behavior.
Compliance requirements can also affect performance. Logging, data residency, encryption, content filtering, and audit controls add processing and infrastructure overhead. Excluding them may produce a benchmark that cannot be reproduced in production.
GPU utilization indicates how busy an accelerator is, but it does not directly measure service quality. Low utilization may indicate underfilled batches or low traffic. High utilization may indicate efficient execution—or a saturated system with growing queues.
GPU utilization should therefore be analyzed alongside:
- TTFT and P99 latency
- Memory use
- Queue depth
- Batch size
- Requests per second
- Output token rate
- Error and timeout rates
A delivery truck with a full cargo load is efficient only if it still arrives on time. Similarly, a fully utilized GPU is not automatically a successful serving system.
Larger batches often improve hardware efficiency by creating more parallel work. However, they can increase waiting time and memory consumption. High concurrency may increase throughput until the system reaches a bottleneck, after which latency rises sharply.
KV-cache usage is particularly important for long-context workloads. A system may perform well with short prompts but degrade rapidly as context length and active sequences increase.
Insight: The Hidden Trade-Offs in LLM Inference Benchmarking
Increasing batch size or concurrency can improve throughput while worsening individual latency. Measure both at multiple load levels instead of selecting only the highest sustainable throughput point.
Quantization, smaller models, caching, and shorter outputs can reduce cost. Each may also affect answer quality, consistency, or task completion. Cost per token is meaningful only when paired with a quality target.
Pushing an accelerator near saturation may improve utilization and reduce idle cost. It also leaves less capacity for bursts and increases the risk of queue buildup. A reliable system often needs headroom.
An optimization that improves the mean may worsen P99. Report distributions and investigate outliers rather than relying on one summary number.
A fixed prompt and constant request rate make testing easy to reproduce. Real traffic includes bursts, retries, long contexts, short responses, malformed requests, and changing workloads. Use simple tests for diagnosis and realistic tests for deployment decisions.
At minimum, calculate cost per request, cost per input token, and cost per output token. For mixed workloads, also calculate cost per successful task or completed user interaction.
A useful formula is:
Cost per request = total serving cost / completed requests
For quality-aware analysis:
Cost per successful task = total serving cost / tasks meeting quality and latency targets
Infrastructure cost includes more than accelerator rental. Include host servers, storage, networking, orchestration, observability, cooling, power, and idle capacity.
Power measurements can be reported as watts, watt-hours per request, or energy per million tokens. These values should be collected under the same workload because power varies with utilization and model behavior.
Long prompts increase prefill computation and memory use. Long completions increase decode time and hold resources for longer. A benchmark using short prompts and short answers may significantly underestimate production cost.
For this reason, report prompt and output length distributions, not only their averages.
Use a fixed test set for controlled comparisons, then add representative traffic patterns for deployment realism. Test sequential requests, steady concurrency, ramp-up loads, and bursty arrivals.
Document model versions, precision, quantization, context limits, sampling parameters, batching, parallelism, and caching. If multiple variables change together, label the result as a system-level comparison rather than attributing it to one optimization.
Requests per second measures completed requests. Tokens per second measures token volume. TTFT measures initial responsiveness. TPOT measures streaming behavior. Goodput measures compliant completions.
No single metric replaces the others. Together, they reveal whether a system is fast, productive, and dependable.
The following comparison shows why these metrics should not be treated as interchangeable.
| Metric | Measures | Best for | Main limitation |
|---|---|---|---|
| RPS | Completed requests | Capacity | Length-sensitive |
| TPS | Processed tokens | Compute output | Can hide waits |
| TTFT | First-token delay | Interaction | Ignores completion |
| E2E latency | Full response time | User completion | Depends on length |
| Goodput | Compliant requests | SLO planning | Requires targets |
The key takeaway is that throughput metrics describe system capacity, while experience metrics describe how the service feels and whether it meets operational commitments.
Forecast: What Better LLM Inference Benchmarks Will Measure
Future benchmarks will increasingly use scorecards rather than one ranking. A scorecard may show cost, median and tail latency, quality, energy, utilization, and goodput at several concurrency levels.
This approach reflects how organizations actually choose systems. The fastest model may not be the cheapest. The cheapest may not meet quality requirements. The most efficient system may lack enough capacity for traffic spikes.
Energy efficiency will become more important as inference workloads expand. Organizations will measure watt-hours per request alongside cost and latency.
The strongest evaluations will connect these variables. For example, reducing energy per token is valuable only if it does not cause unacceptable quality degradation or reliability problems.
Static benchmarks become less representative as models, prompts, and application behavior change. Adaptive tests will adjust traffic based on observed queueing, context length, and output distribution.
Interactive chat prioritizes TTFT and smooth streaming. Batch workloads prioritize total throughput and cost. Agentic workloads emphasize repeated calls, tool latency, structured-output validity, and failure recovery.
Each workload requires a different weighting of metrics.
Future evaluations will include traffic bursts, long-context prompts, variable completion lengths, retries, and partial failures. These conditions expose bottlenecks that steady-state tests often miss.
Call to Action: Build a More Reliable Benchmarking Process
Start by defining what success means. Set targets for response quality, TTFT, end-to-end latency, error rate, availability, cost, and energy. Separate hard requirements from optimization goals.
At minimum, measure:
- Cost per request and successful task
- Input and output tokens
- TTFT, TPOT, and end-to-end latency
- Median, P95, and P99 values
- RPS, TPS, and goodput
- Error, timeout, and retry rates
- GPU utilization and memory consumption
Use the same workload files, warm-up procedure, duration, load profile, and measurement boundaries for every comparison.
Document accelerator type, device count, host configuration, model version, precision, serving engine, driver, batching policy, concurrency, prompt lengths, output limits, and traffic pattern.
Do not report only peak TPS. Include latency percentiles, goodput, error rates, GPU utilization, and memory behavior. Explain how each metric was calculated.
Repeat tests across multiple runs and load levels. Check whether results are stable and whether the benchmark reflects production traffic. Investigate surprising gains, especially when a system appears faster but produces shorter outputs or fails more requests.
A deployment decision should be based on the operating point that meets quality and reliability targets at an acceptable cost—not on the most impressive isolated number.
Conclusion: Measure the Trade-Offs Behind LLM Performance
LLM inference benchmarking is most useful when it answers operational questions: How quickly will users see a response? What will each successful task cost? How much traffic can the system handle? What happens during bursts? Which requests fail the service target?
Answering those questions requires more than token throughput. It requires a complete view of prefill, decode, queueing, latency distributions, quality, resource consumption, and failures.
The best inference system is rarely the one that wins one metric. It is the system that delivers the required quality with acceptable latency, sustainable cost, and predictable reliability.
As LLM applications move from prototypes to production, benchmarks will become more workload-specific, multi-objective, and energy-aware. Teams that measure these trade-offs early will make better choices about models, GPUs, serving architectures, and capacity planning—and avoid optimizing a number that users never experience.