How to Benchmark LLM Latency and Throughput in Production

Introduction

Learning how to benchmark LLM latency and throughput in production separates teams that guess about capacity from teams that plan it. Google researchers wrote in The Tail at Scale that systems responding within about 100 milliseconds feel fluid, and that rare latency spikes come to dominate performance as services grow. Language model serving inherits that lesson, because one streaming reply involves queuing, prefill, many decode steps, and several network hops. A single average number therefore hides most of what users actually feel. This guide explains the metrics, the workloads, the tools, and the statistics that turn raw timings into decisions you can defend. It also walks through a seven step runbook, real examples from teams that serve models at scale, and the mistakes that quietly corrupt results.

Quick Answers on Benchmarking LLM Latency and Throughput

How to benchmark LLM latency and throughput in production without fooling yourself?

Replay production-shaped prompts against your real serving stack, sweep concurrency, and record time to first token, inter-token latency, end-to-end latency, and tokens per second as p50, p95, and p99 after warm-up.

Which metrics matter most for LLM latency in production?

Time to first token captures responsiveness, inter-token latency captures streaming smoothness, and end-to-end latency captures total wait. Report each as percentiles, because averages hide the slow requests users remember.

What is goodput in LLM benchmarking?

Goodput is the rate of requests that meet every latency objective you set, such as limits on time to first token and time per output token. It shows usable capacity, while raw throughput can look healthy during overload.

Key Takeaways on Benchmarking LLM Latency and Throughput

• Report time to first token, inter-token latency, and end-to-end latency as percentiles, never as one average.
• Sweep concurrency from a single request to just above your maximum batch size to find where throughput plateaus and latency climbs.
• Describe usable capacity with goodput against explicit service level objectives instead of raw tokens per second.
• Warm up, repeat runs, fix token lengths, and control caching so a result stays comparable from one release to the next.

What Is LLM Latency and Throughput Benchmarking

How to benchmark LLM latency and throughput in production starts with defining it: replaying realistic traffic against a live serving stack, then recording time to first token, inter-token latency, end-to-end latency, and tokens per second as percentiles under controlled concurrency.

Why Production Benchmarks Differ From Vendor Leaderboards

Public benchmarks exist to compare hardware and models under tightly controlled conditions, and that purpose differs sharply from planning capacity for your own traffic. The MLCommons documentation for its Llama 2 70B benchmark explains that the server scenario target is usually around 80 percent of the offline rate. On some systems that figure falls below 50 percent, and that gap shows how much sustainable throughput shrinks once a latency constraint is enforced on every request. Vendors often quote the unconstrained figure because it looks best on a slide. Your users never experience that figure, since they only feel the latency of their own request. Treat every leaderboard number as an upper bound that your own harness must confirm.

Your production traffic has features that no public suite reproduces, and each one moves the numbers. Prompts arrive with a messy mix of lengths, system prompts repeat across thousands of sessions, and users abandon streams halfway through a reply. Traffic also comes in bursts, so the queue in front of your GPUs grows and shrinks throughout the day. A benchmark that ignores these patterns will report a comfortable margin that disappears the first time a marketing email lands. Cost planning depends on the same realism, as the guide on reducing LLM inference costs shows when it traces spend back to utilization.

A useful production benchmark answers three concrete questions before anyone opens a dashboard. First, how many concurrent users can one replica serve while still meeting the latency objectives you promised. Second, what time to first token and per-token delay do users see at your expected peak, expressed as tail percentiles. Third, what changes after a model upgrade, a new quantization, or a different batching configuration. That is why learning how to benchmark LLM latency and throughput in production begins with questions, not tools. Once the questions are written down, the metrics, workload, and load pattern almost choose themselves.

How Latency Is Measured: Time to First Token and Beyond

Shifting from planning to measurement, the first task is agreeing on what each latency number means. NVIDIA’s benchmarking metrics reference defines time to first token as the delay between submitting a query and receiving the first token. It notes that this delay includes queuing, prefill, and network time. Inter-token latency is the average gap between consecutive output tokens after that first one. End-to-end latency is simply time to first token plus the generation time that follows. Keeping those three apart lets you tell a slow start from a slow stream, which have very different causes.

Different tools disagree on the details, and the disagreement can quietly ruin a comparison. The NVIDIA reference computes inter-token latency as end-to-end latency minus time to first token, divided by the number of output tokens minus one. It also warns that some tools fold the first token into the average, so you should compare numbers only when definitions match. Time per output token, usually shortened to TPOT, is treated as the same quantity in several tools. Write the formula you used into your benchmark report so that a colleague can reproduce it months later.

Latency in a streaming system is dominated by output length, which surprises many teams. A Databricks engineering analysis of LLM inference performance expresses total latency as time to first token plus time per output token multiplied by the number of tokens generated. Their data showed that adding 512 input tokens cost less latency than generating eight extra output tokens on the models they tested. A time per output token of 100 milliseconds equals ten tokens per second per user, roughly 450 words per minute. That is already faster than most people read, so chasing lower values pays off only for voice, agents, or batch pipelines.

Different products care about different parts of the request timeline and the user experience. A chat assistant lives or dies on time to first token, because a blank screen feels broken even when the full answer arrives quickly. A code completion tool needs both a fast start and a tight end-to-end budget, since suggestions that arrive late are discarded. A document summarizer with a long prompt can accept a slower start but needs steady streaming. An agent that chains many model calls multiplies every delay, which is why agent evaluations start with per-call latency. Choose the metric that matches the experience before you choose a number to defend.

Throughput, Goodput, and the Capacity Curve

Building on those latency definitions, throughput describes how much work the system completes rather than how fast one request feels. The NVIDIA reference measures total tokens per second as all output tokens across simultaneous requests divided by the span from the first request to the last response. Requests per second divides completed requests by the same window. It also observes that total throughput rises with concurrency until GPU resources saturate, after which it can flatten or even decline. Meanwhile the speed each individual user experiences keeps falling as latency rises.

Raw throughput is easy to inflate, which is why goodput deserves a place in every report. The DistServe paper defines goodput as the maximum request rate a GPU can sustain while meeting both the time to first token and time per output token constraints. Its authors reported serving up to 7.4 times more requests, or meeting up to 12.6 times tighter service level objectives, than state-of-the-art systems. More than 90 percent of requests stayed within their latency constraints. You can read the full claims in the DistServe paper on arXiv. The lesson for benchmarkers is that capacity only counts when the latency promise holds.

Plotting a capacity curve turns those definitions into a planning tool. Run the same workload at rising concurrency levels, and for each level record tokens per second, goodput, and the p95 and p99 of each latency metric. The curve usually shows a knee where throughput stops growing but tail latency keeps climbing. Operate comfortably below that knee, because a system running at the edge of saturation has no room for bursts. The resulting requests-per-GPU figure feeds directly into budgets, as the analysis of generative AI cost per token explains for finance teams.

Designing a Workload That Looks Like Your Real Traffic

Beyond the metrics, a benchmark is only as honest as the prompts it replays. NVIDIA’s guide to LLM benchmarking fundamentals lists typical shapes for common tasks. Translation runs roughly 500 to 2,000 tokens in and out, while generation uses about 100 tokens in and 1,000 out. Summarization or retrieval flips that to about 1,000 in and 100 out, and reasoning uses around 100 in with up to 10,000 out. These shapes stress the system in opposite ways, since long inputs strain prefill and long outputs strain decoding. Sample the length distribution from your own logs instead of choosing one fixed size. A single mean length hides the long-tail requests that dominate your p99.

Fidelity also depends on the details around the prompt text. Count tokens with the tokenizer your model actually uses, because token counts differ between tokenizers and change both cost and timing, as the explainer on tokenization in LLMs shows. Keep streaming enabled if production streams, include the real system prompt, and match sampling settings. When you need identical output lengths for a controlled comparison, force generation to ignore the end-of-sequence token, but label that run as synthetic. Redact personal data before replaying anything captured from users, and store the sanitized set so every future run uses exactly the same prompts.

Load Generators and Benchmark Tools Worth Knowing

Choosing among the available tools starts with the serving engine you run, because the closest match usually reports metrics the same way the engine does. The vLLM project ships vllm bench serve, documented in its benchmark serving reference, which drives a running OpenAI-compatible server and reports throughput plus latency statistics. It supports a target request rate sampled from a Poisson process, a burstiness setting, a ramp-up mode, and a cap on in-flight requests. A goodput option accepts objectives for time to first token, time per output token, and end-to-end latency. A request counts as good only if it meets every listed objective. Results can be saved as JSON, which makes regression comparisons straightforward.

NVIDIA’s tooling approaches the same problem from the inference platform side. GenAI-Perf keeps a fixed number of requests active throughout a run, which makes concurrency the primary knob. It uses a sliding window to exclude warm-up and cool-down requests, and NVIDIA documents the metrics of its AIPerf tool formula by formula. The Ray project’s LLMPerf took a different route, sending requests in batches and measuring against many hosted providers through one interface. Its repository README lists support for OpenAI-compatible APIs, Anthropic, TogetherAI, Hugging Face, LiteLLM, Vertex AI, and SageMaker. The owner archived the repository on December 17, 2025, so treat it as a reference design rather than a maintained dependency.

No tool replaces thinking about what it does to your numbers. The NVIDIA fundamentals guide found that LLMPerf divides token counts by the full benchmark duration. Prompt generation and response storage consumed about a third of the run in single-concurrency tests. GenAI-Perf divides by the span from the first request to the last response instead, so the same server can look slower under one tool. The LLMPerf README adds that results depend on the provider backend, vary with time of day and load, and may not match your actual workload. Pick one primary tool, document its definitions, and cross-check a few runs with a small client you control.

Concurrency, Arrival Patterns, and Steady State

From there, the shape of the load matters as much as the tool that generates it. NVIDIA’s guide recommends driving tests by concurrency, meaning N users each holding one active request, because a fixed request rate above system throughput lets outstanding requests grow without bound. When concurrency exceeds the maximum batch size multiplied by the number of active replicas, requests start to queue and time to first token rises sharply. Sweep from one request up to a value slightly above that product so the whole curve is visible. Add an open-loop run with Poisson or bursty arrivals afterward, since real users do not wait politely for the previous reply.

A result only means something once the system reaches steady state. The Anyscale benchmarking guide recommends a warm-up phase, repeated runs that you average, and a written record of GPU type, replica count, and software versions. It also warns that autoscaling can add replicas mid-run and distort a single-replica measurement, so pin the replica count when you want per-replica numbers. Cold starts and cache misses pollute the first requests, and prefix caching can inflate results when prompts share long prefixes that you did not intend to share. Decide deliberately whether caching belongs in the baseline, using the overview of prompt caching and LLM costs to understand what it changes.

Percentiles, Tail Latency, and Statistical Honesty

Stepping back from tools, the statistics you report decide whether anyone can trust the result. Google’s SRE book chapter on monitoring distributed systems offers a memorable example. A service averages 100 milliseconds at 1,000 requests per second, yet 1 percent of requests could still take five seconds. That slow tail is 50 times the average, yet the average alone would look perfectly healthy on a dashboard. The chapter adds that when a page depends on several backends, one backend’s 99th percentile can easily become the median experience of the frontend. LLM serving stacks have exactly this structure, with queues, schedulers, and multi-call agents stacked on top of each other.

Report percentiles for every latency metric separately instead of blending them. Time to first token, inter-token latency, and end-to-end latency each deserve their own p50, p95, and p99. A stable p50 can sit next to a terrible p99 on any one of them. The SRE chapter recommends histogram buckets that grow roughly by a factor of three. Example boundaries are 0 to 10 milliseconds, 10 to 30, 30 to 100, and 100 to 300. Track failed and timed-out requests separately, since a fast error can mask a problem and a slow error is worse. Teams that already follow the practices in measuring AI agent performance will recognize the same discipline of splitting success from latency.

Sample size is the quiet trap behind many tail numbers. A p99 computed from 1,000 requests rests on only the ten slowest observations, so one noisy neighbor or garbage collection pause can move it dramatically. Run long enough to collect several thousand requests per concurrency level, and repeat the run to see how much the percentile itself wobbles. The SRE chapter also suggests that the 99th percentile over a short window, such as one minute, works as an early saturation signal in production. That makes the same metric useful in both your benchmark and your live alerting, which keeps the two views consistent.

Hardware, Batching, and Memory Bandwidth Effects

Turning to the hardware underneath, LLM decoding behaves differently from the compute-heavy workloads most engineers know. The Databricks analysis of LLM inference performance engineering explains that at small batch sizes decoding is limited by how fast model parameters move out of GPU memory. It defines memory bandwidth utilization as achieved bandwidth divided by peak bandwidth, where achieved bandwidth equals parameter size plus KV cache size divided by time per output token. A 7 billion parameter model in 16-bit precision with a 14 millisecond token time moves about 14 GB in 14 milliseconds, or roughly 1 TB per second. On a machine with 2 TB per second of bandwidth, that is 50 percent utilization. Prefill, by contrast, is generally compute-bound, which is why long prompts mostly affect time to first token.

Batch size is the main lever that trades latency for throughput. In the Databricks tests of MPT-7B with static batching on one A100, using 512 input and 64 output tokens, throughput rose steeply. It climbed from 0.9 requests per second at batch size one to 8.0 at batch 16 and 12.5 at batch 64. Pushing to batch 64 raised latency roughly fourfold while throughput rose roughly fourteenfold. The same article notes that continuous batching can deliver 10 to 20 times better throughput than dynamic batching. Your benchmark should therefore record the batching configuration alongside every number, because the same GPU can look slow or fast depending on that one setting.

Adding GPUs through tensor parallelism helps, but the gains are uneven and depend on the model. For Llama 2 70B with a 512 token input at batch size one, Databricks measured 154 milliseconds to first token on four A100-40GB GPUs. Doubling the hardware to eight GPUs brought that figure down to 114 milliseconds. That is a 0.74 ratio for doubling the hardware, not a halving. Going from two to four GPUs for MPT-7B cut token latency about 12 percent at batch size one but about 33 percent at batch size 16. Smaller models lose bandwidth efficiency faster as GPU counts rise, so the best parallelism setting for one model is rarely right for another.

Newer silicon can move the numbers a lot, yet vendor claims still need local confirmation. The H100-80GB has 2.15 times the memory bandwidth of the A100-40GB. Databricks reported that this cut Llama 2 70B latency on four-GPU systems by 36 percent at batch size one and 52 percent at batch size 16. The article also observed that two cloud providers’ eight-GPU A100 servers differed by a factor of two in latency. That spread shows why you should benchmark on the exact instance type you plan to rent. For a broader view of the accelerator landscape, see the comparison of GPUs, TPUs, and NPUs before you commit to a purchase.

Putting a Benchmark Into Your Release Pipeline

On top of one-off studies, the biggest return comes from running the same benchmark automatically whenever something changes. Store a baseline result file in version control, and run the harness on every model version, engine upgrade, or batching change. Then compare each new run against the stored baseline result file. Fail the build when goodput drops below the agreed fraction or when p99 time to first token exceeds the objective. Record GPU type, replica count, and software versions with each result, as the Anyscale guide recommends, so a difference can be traced to its cause. Keep the prompt set frozen and versioned, because a quietly changed dataset makes trend lines meaningless.

Speed gains are only useful if answer quality survives them. Quantization, speculative decoding, and aggressive batching can all alter outputs, so pair each performance run with a small quality evaluation on the same release candidate. A scoring pass built on evaluating LLM outputs with a judge model can flag regressions that latency charts never show. Treat the pair of results as one gate, and require both to pass before a change reaches production. A faster model that quietly degrades answers simply moves the cost from your GPU bill to your support queue.

Finally, translate throughput into money so the benchmark informs budgets, not only dashboards. Dividing the hourly price of a replica by the tokens it sustains within your objectives gives a cost per million tokens that finance teams can compare with API pricing. Share that number together with the latency percentiles, because a cheaper configuration that violates the objectives is not cheaper in any useful sense. The playbook on enterprise AI cost optimization strategies shows how such unit costs feed larger planning decisions. Review the figures quarterly, since hardware prices and engine performance both shift quickly.

Risks That Produce Misleading Benchmark Numbers

Despite the care that goes into a harness, a handful of familiar traps produce numbers that look precise and are wrong. Forcing every response to a fixed length is useful for controlled comparisons, and NVIDIA’s fundamentals guide advises ignoring the end-of-sequence token for that reason. The same trick makes output lengths unrealistically uniform, so batching looks tidier than it will in production. Running the load generator on the same machine as the server lets the two compete for CPU and network, which understates throughput. Measuring only on the server side hides network and gateway delay that users still experience. Measure from the client, and keep a server-side view as a cross-check.

Statistical and configuration mistakes cause the rest of the damage. A single run on a shared cloud instance can land on a noisy neighbor, so one unlucky result may be mistaken for a regression. Comparing a quantized model with a full-precision one is only fair when you also compare answer quality, as the overview of post-training quantization for edge AI makes clear. Mixing tokenizers, engine versions, or sampling settings between runs breaks comparability without any visible warning. Reporting only the mean, or only the best of several runs, turns an honest measurement into marketing. A short pre-flight checklist covering workload, warm-up, caching, replicas, and versions catches most of these problems before a long run starts.

Ethics and Environmental Cost of Inference Benchmarks

Looking beyond engineering, benchmarks carry responsibilities that rarely appear in a runbook. Honest reporting is the first one, because a number published without its workload, hardware, and percentile invites decisions its authors never intended. Cherry-picking the best concurrency level or the friendliest prompt mix may win an internal argument, yet it misleads colleagues who size budgets from the result. Disclose the configuration, the caching state, and the run count whenever a figure leaves your team. If a vendor shares a figure without those details, ask for them before it enters a business case.

Privacy is the second responsibility, since the most realistic prompts come from real users. Production logs can contain names, health details, source code, and credentials, and a benchmark dataset copied to a laptop multiplies that exposure. Redact personal data before sampling, limit who can read the prompt set, set a retention date, and document the legal basis for reusing the data. Synthetic prompts that mimic the length and structure of real traffic are a safer substitute when policy forbids reuse. Treat the stored benchmark corpus with the same care as any other copy of customer data.

Energy use and access to fair measurement complete the picture for responsible teams. Every sweep burns GPU hours, so right-size the run, reuse results across experiments, and avoid repeating an expensive sweep that a smaller one would answer. The environmental stakes of inference are growing, as the discussion of generative AI energy costs explains for data centers and power grids. Tokenizers also split some languages into more tokens than others, so latency and price can differ for users who write in those languages. Include multilingual prompts in your sample so that the numbers describe all of your customers, not only the easiest ones.

The Future of LLM Latency and Throughput Benchmarking

Looking ahead, service level objectives will likely replace raw tokens per second as the headline metric. Goodput already gives a way to express capacity under a latency promise, and research systems such as DistServe optimize directly for it. Separating prefill and decoding onto different GPUs is the idea behind that work. Each phase then needs its own benchmark, with time to first token tuned for one pool and time per output token for the other. Expect load generators to report per-phase goodput as a standard output within the next few years. Teams that already track percentiles and objectives will adopt that change with little friction.

Workloads are shifting too, and benchmarks must follow them to stay relevant for real users. NVIDIA’s reasoning profile pairs short inputs with outputs of 1,000 to 10,000 tokens, which moves nearly all cost into decoding and makes inter-token latency the dominant concern. Agents multiply the effect by chaining many model calls, so a task-level latency and cost budget becomes more useful than any single request metric. Standardized suites such as MLPerf will keep adding scenarios, but they will still describe typical conditions rather than yours. Your own harness remains the only source of numbers that reflect your prompts, your hardware, and your users.

Deployment locations are diversifying as well, which complicates how teams compare results. Smaller models on phones, laptops, and local servers now handle work that once required a data center. Hybrid designs route each request to the cheapest place that meets its objective. The comparison of edge AI and cloud AI outlines the trade-offs that routing logic must weigh. Benchmarks will need to cover both ends and the handoff between them, including the network. Whatever the hardware, how to benchmark LLM latency and throughput in production will keep coming down to realistic workloads, honest percentiles, and repeatable runs.

How to Run a Production LLM Benchmark in Seven Steps

This runbook uses a vLLM server as the example target, yet the same structure works with any serving engine and any load generator. Each step produces an artifact that the next step consumes, so you can automate the whole sequence later. Expect the first pass to take a day, mostly spent on capturing a clean prompt set. Later passes take only minutes because the prompt set and scripts already exist. Follow the order, since skipping the objectives or the warm-up makes every later number harder to interpret.

Step 1 – Write down your service level objectives

Start with the experience you want to protect, and translate it into percentile targets with a time window. A chat product might require a p95 time to first token under 1.5 seconds. It might also require a p95 time per output token under 100 milliseconds, which equals ten tokens per second per user. An end-to-end budget for a typical answer length gives the third objective. These values are examples, so replace them with numbers your product team has agreed to. Write each objective as a percentile target, never as an average, so the benchmark can compute goodput against it.

{
  "ttft_p95_ms": 1500,
  "tpot_p95_ms": 100,
  "e2e_p95_ms": 15000,
  "min_success_rate": 0.99
}

Step 2 – Capture a representative prompt set

Sample prompts from production logs so the length distribution matches reality, and stratify by feature or tenant if traffic differs widely. Redact personal data before the sample leaves the logging system, and store the sanitized file in version control or a controlled bucket. Count input tokens with the model’s own tokenizer, and record the observed output lengths for the same requests. Aim for at least 3,000 prompts so that tail percentiles rest on enough observations. Save the result as a JSONL file where each line holds one prompt and its expected output length.

import json, random

def write_prompt_set(rows, path, n=3000, seed=7):
    random.seed(seed)
    sample = random.sample(rows, min(n, len(rows)))
    with open(path, "w") as f:
        for r in sample:
            f.write(json.dumps({"prompt": r["prompt"], "output_len": r["output_len"]}) + "\n")

Step 3 – Stand up an isolated target and warm it up

Deploy the model on hardware identical to production, with the same engine version, quantization, and parallelism settings. Pin the replica count at 1 so that autoscaling cannot add capacity in the middle of a run, and keep other jobs off the machine. Developers who want a quick local rehearsal can follow the guide to running open source LLMs locally before touching shared GPUs. Send about 200 throwaway requests after startup so that kernels compile, caches fill, and memory settles. Discard those requests from the measured results so that startup effects do not skew percentiles.

export MODEL="your-org/your-model"
vllm serve "$MODEL" --port 8000

Step 4 – Run a concurrency sweep

Run the same prompt set at rising concurrency, such as 1, 2, 4, 8, 16, 32, and 64, stopping slightly above the maximum batch size of your engine. For each level record throughput, goodput, and the p50, p95, and p99 of time to first token, time per output token, inter-token latency, and end-to-end latency. Run each level for at least a few minutes, or a few thousand requests, so the tail is stable. The loop below uses the vLLM benchmark client with objectives expressed in milliseconds. Flag names can differ between versions, so confirm them with the help output of your installed release.

for C in 1 2 4 8 16 32 64; do
  vllm bench serve --backend vllm --model "$MODEL" \
    --dataset-name custom --dataset-path prompts.jsonl \
    --num-prompts 2000 --max-concurrency $C \
    --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99 \
    --goodput ttft:1500 tpot:100 e2el:15000 \
    --save-result --result-filename "sweep_c${C}.json"
done

Step 5 – Compute percentiles and goodput yourself

Cross-check the tool output with a small script that reads per-request timings and applies your own definitions. This protects you from silent differences in how tools define inter-token latency or divide tokens by time. The script below treats a request as good only when it succeeded and met all three objectives, which mirrors the goodput definition used in the research literature. Compare its numbers with the tool summary, and investigate any gap larger than 5 percent, since a mismatch usually exposes a definition difference or a logging bug. Keep this script alongside the results so future runs use identical math.

import json
import numpy as np

def summarize(path, slo):
    rows = [json.loads(l) for l in open(path)]
    ok = [r for r in rows if r["ok"]]
    pct = lambda k, q: float(np.percentile([r[k] for r in ok], q))
    good = [r for r in ok if r["ttft_ms"] <= slo["ttft_p95_ms"]
            and r["tpot_ms"] <= slo["tpot_p95_ms"] and r["e2e_ms"] <= slo["e2e_p95_ms"]]
    span = max(r["end_s"] for r in rows) - min(r["start_s"] for r in rows)
    return {"p99_ttft_ms": pct("ttft_ms", 99), "p99_e2e_ms": pct("e2e_ms", 99),
            "tokens_per_s": sum(r["out_tokens"] for r in ok) / span,
            "goodput_rps": len(good) / span}

Step 6 – Add bursts, caching, and failure conditions

Closed-loop sweeps show capacity, but production also brings bursts, so add an open-loop run with a fixed request rate and a burstiness setting below one. Repeat the best configuration with prefix caching on and off, and report both, because the difference tells you how much your traffic shares prefixes. Include a cold-start run and, where possible, a run with 1 replica removed. Newer architectures can shift the whole curve, as DeepSeek’s reported compute savings illustrated, so rerun the sweep after any model change. Record every condition in the result file name so nothing is ambiguous later.

vllm bench serve --backend vllm --model "$MODEL" \
  --dataset-name custom --dataset-path prompts.jsonl \
  --num-prompts 3000 --request-rate 8 --burstiness 0.5 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99 \
  --save-result --result-filename "burst_rate8.json"

Step 7 – Store the baseline and gate every release

Commit the summary of your best run as the baseline, together with the GPU type, replica count, engine version, and prompt set hash. In continuous integration, rerun the benchmark for every candidate release. Fail the build when p99 time to first token exceeds the objective or goodput drops beyond the allowed fraction. Pair the check with a quality evaluation so a speedup never hides a regression in answers. Teams that run agents should add the cost-per-task view from the article on measuring agent cost per task. Review the baseline whenever hardware, traffic mix, or product promises change.

import json, sys

base = json.load(open("baseline.json"))
new = json.load(open("candidate.json"))
if new["p99_ttft_ms"] > 1500 or new["goodput_rps"] < 0.9 * base["goodput_rps"]:
    print("benchmark gate failed")
    sys.exit(1)
print("benchmark gate passed")

Key Insights Engineers Cite on LLM Latency and Throughput

• Google’s Tail at Scale research notes that systems answering within about 100 milliseconds feel fluid, so even rare slow responses become visible to users at large scale.
• Databricks measured MPT-7B throughput climbing from 0.9 to 12.5 requests per second as batch size grew from one to 64, while latency rose roughly fourfold on a single A100 GPU.
• The DistServe authors report serving up to 7.4 times more requests by splitting prefill and decoding across separate GPUs, while keeping over 90 percent of requests within their latency constraints.
• LMSYS cut the GPUs needed for Chatbot Arena traffic by 50 percent after adopting vLLM, which handled an average of 30,000 requests per day in the reported period.
• Anyscale reported up to 23 times higher throughput than naive static batching when combining continuous batching with vLLM, measured on one A100 40GB GPU running OPT-13B.
• Character.AI says it serves about 20,000 queries per second and has reduced serving cost at least 33 times since its 2022 launch, though these are unaudited company claims.
• Google’s SRE guidance shows a service with a 100 millisecond average can still have 1 percent of requests taking five seconds, which is 50 times the mean latency.

Taken together, these findings point to one habit: judge a serving stack by tail percentiles and goodput under a latency promise, not by peak tokens per second. Batching and continuous scheduling deliver large throughput gains, yet every gain trades against the delay each user sees. Separating prefill from decoding shows that the two phases deserve independent targets and independent measurements. Company claims and research results use different workloads and hardware, so they inform direction rather than replace your own numbers. Treat each reported multiple as an upper bound, and let your own harness decide how much of it survives contact with your traffic.

Comparing the Main LLM Benchmarking Tools Side by Side

Given the differences in definitions and load control, a side-by-side view helps explain why identical servers produce different headline numbers. The table highlights definitions, load control, and maintenance status, because those three differences explain most cross-tool disagreement. Each cell summarizes what the vLLM documentation, NVIDIA’s metrics reference, and the LLMPerf repository say about their own behavior. Where a page does not document a feature, the table says so instead of guessing. Use it as a checklist of questions to ask before trusting any number a tool prints. Pick the tool that matches your engine, then verify its definitions against the formulas you wrote down.

Dimension	vllm bench serve	GenAI-Perf and AIPerf	LLMPerf
Best fit	Self-hosted vLLM servers	NVIDIA-based inference stacks	Comparing hosted API providers
Load control	Request rate, burstiness, ramp-up, and a concurrency cap	A fixed number of concurrent requests held active	Concurrent requests sent in batches that drain
Inter-token definition	Time per output token excludes the first token	Excludes time to first token from the average	Includes time to first token in the average
Throughput denominator	Reports output, total, and peak token throughput	First request to last response	Full benchmark duration with overheads
Goodput support	Yes, through latency objectives	Not defined on the metrics page	Not documented in the README
Percentile reporting	Configurable list, default P99	Not covered in the cited pages	Summary and per-request metrics files
Warm-up handling	Warm-up requests supported	Sliding window excludes warm-up and cool-down	Not mentioned in the README
Maintenance status	Ships with vLLM releases	Documented by NVIDIA	Archived December 17, 2025

Benchmarking Examples From Teams Serving LLMs at Scale

LMSYS Chatbot Arena Moves Its Backend to vLLM

LMSYS adopted vLLM as the backend for Chatbot Arena after peak traffic grew several times over during spring 2023. According to the vLLM launch post, the team deployed the engine and cut the number of GPUs needed for the same traffic by 50 percent. An internal micro-benchmark showed up to 30 times higher throughput than the initial Hugging Face backend. The service handled an average of 30,000 requests per day, with peaks of 60,000. The engine reduced KV cache waste from 60 to 80 percent in existing systems to under 4 percent through paged memory management. The 30 times figure still comes from an internal micro-benchmark rather than a reproducible public comparison, so treat it as a best case. Results also covered only two model and GPU pairings, and the post dates from June 2023, so current baselines are stronger.

Anyscale Compares Batching Strategies on OPT-13B

Anyscale ran a controlled comparison of batching strategies on a single NVIDIA A100 40GB GPU serving Meta’s OPT-13B model. The team sent 1,000 requests with 512 input tokens, drew output lengths from an exponential distribution with a mean of 128, and measured naive static batching against continuous batching. According to the Anyscale write-up, continuous batching alone produced up to 8 times higher throughput, and vLLM’s memory optimizations pushed the increase to as much as 23 times. Median latency improved as well, with gains across percentiles at one and four queries per second. The workload was synthetic because the model was forced to ignore end-of-sequence tokens, which limits how far the numbers transfer. Anyscale also sells Ray-based services, and the authors called the saturation point anecdotal and said an apples-to-apples comparison needs more experimentation.

Perplexity Tunes Llama 2 70B on H100 GPUs

Perplexity ran Llama 2 70B on eight-GPU nodes with NVIDIA TensorRT-LLM version 0.5.0, using 1,024 input and 512 output tokens per request to choose between A100 and H100 configurations. In its engineering write-up, the same configuration on H100 reached 54 percent of the A100 latency with 184 percent of the throughput. Switching the H100 from fp16 to fp8 cut latency by roughly 10 percent and changed WikiText perplexity by less than 1 percent. In contrast, 4-bit weights shifted perplexity by about 7 percent. A tensor-parallel-2 setup at batch size 128 delivered 767 output tokens per second per GPU, which is 460 percent of the A100 baseline. That setup also doubled latency to about 42 seconds for 512 tokens, so the authors called it probably unsuitable for production. The tests used a synthetic dataset, one model, and one software version, which limits how far the conclusions generalize.

Lessons From Case Studies in Production LLM Serving

Case Study: Character.AI Serving 20,000 Queries per Second

Character.AI faced a serving problem unusual for a consumer startup, because users hold long open-ended conversations and every reply must be fast while each conversation hour costs almost nothing. The company says it built its own serving stack and optimized both the Transformer architecture and the attention KV cache, including caching between conversation turns. In its engineering post, it reports serving about 20,000 queries per second, which it compares with roughly 20 percent of the request volume Google Search handles. It also states that serving cost has fallen at least 33 times since the 2022 launch and now sits below one cent per hour of conversation. The post describes the work as a mix of architecture changes and serving optimizations rather than one trick. Caching between turns matters because each new message otherwise forces the system to reprocess the entire earlier conversation.

The post goes further and claims that its traffic costs 13.5 times less than it would on the most efficient leading commercial APIs. A hypothetical scenario of 100 million daily users spending an hour each puts annual serving cost near 365 million dollars. These figures are company claims without independent verification, and the Google comparison relies on public sources the post does not cite. The summary also omits cache hit rates and the detailed architecture changes, so outsiders cannot reproduce the result. Still, the case shows why a serving team measures cost per conversation hour alongside latency percentiles.

Case Study: DistServe and the Goodput Metric

The authors of DistServe identified a bottleneck in how common systems serve models. Prefill and decoding run together on shared GPUs, so each phase interferes with the other and their parallelism choices are tied together. Their solution assigns each phase to separate GPUs and tunes resources and parallelism for each phase against the application's latency targets. It also places the phases using cluster bandwidth to limit the extra communication that the split creates. Latency is measured with time to first token for prefill and time per output token for decoding, and goodput counts only requests that satisfy both. The DistServe abstract reports serving up to 7.4 times more requests or meeting up to 12.6 times tighter latency objectives. More than 90 percent of requests stayed within their constraints across the models, applications, and latency requirements tested.

Those results come from a research system evaluated on selected models and workloads, and the abstract frames them as upper bounds. Disaggregation adds a transfer of cached state between GPU pools, which is a real cost that depends on network bandwidth and placement. Smaller deployments with a handful of GPUs may find the added complexity outweighs the benefit. The case still teaches a lasting benchmarking lesson, because measuring prefill and decoding separately exposes problems that a single latency number would blur. Teams adopting the idea should reproduce the comparison on their own prompts and hardware.

Case Study: Databricks and Memory Bandwidth Utilization

Databricks engineers faced a diagnostic problem, because slow token generation could reflect a hardware limit or an inefficient serving stack that raw tokens per second cannot distinguish. They introduced memory bandwidth utilization, defined as achieved bandwidth divided by peak bandwidth, and measured it across batch sizes, GPU counts, and models in the MPT and Llama 2 families. In their inference performance analysis, two H100-80GB GPUs reached 60 percent utilization at batch size one while four A100-40GB GPUs reached 55 percent. The same study showed utilization falling as batch size grew, because the workload shifts toward being compute-bound. That insight told them where extra hardware would help and where larger batches would simply add delay.

The analysis has several limits that readers should respect before applying its numbers. It covers 2023-era models and GPUs, a specific serving engine, and mostly small batch sizes, so today's engines and chips will produce different absolute values. It also does not discuss tail latency percentiles, which means teams still need to measure p95 and p99 on their own traffic. A utilization figure is a diagnostic rather than a goal, since the best configuration is the one that meets your objectives at the lowest cost. Used that way, the metric turns a vague sense of slowness into a concrete, testable hypothesis.

Frequently Asked Questions on Benchmarking LLM Latency and Throughput