Google Cloud benchmark shows LLM throughput on TPU v6e depends heavily on workload shape
Classification and generation stress accelerators differently, a lesson in why prefill and decode phases should drive model-serving decisions.
Google Cloud has published a benchmarking study comparing how two sizes of its Gemma 3 model behave on TPU v6e when serving two structurally different kinds of request. The study, released this week on the company's developers and practitioners blog, ran Gemma 3 12B and Gemma 3 27B through vLLM using the tpu-inference plugin on a 2x2 chip topology, deployed on Google Kubernetes Engine Autopilot.
The team defined a generation workload with roughly 500 input tokens and 1,000 output tokens, and a classification workload with about 4,000 input tokens and only 10 output tokens. Throughput was normalised to the 12B model at 16 concurrent users and measured at 16, 32, 64 and 128 users.
For generation, the smaller model scaled far better. Gemma 3 12B reached 8.19 times baseline throughput at 128 users, while the 27B model plateaued at 4.12 times, hitting what the authors call a performance wall. For classification the gap almost disappeared: 12B reached 6.37 times and 27B 6.04 times at the same concurrency. End-to-end latency for the 27B model rose to 3.33 times baseline on generation and 3.88 times on classification at 128 users.
The post recommends deploying the 12B model for high-concurrency generation, capping the 27B model at 64 concurrent requests per replica for that workload, and using the larger model freely for prefill-heavy classification. It advises scaling replicas on end-to-end latency rather than CPU or memory signals, setting the VLLM_TPU_BUCKET_PADDING_GAP variable to keep scaling linear, and using a standard serving configuration of a 128,000-token maximum model length, 8,192 maximum batched tokens and 512 maximum sequences. It also warns that at 128 users, misconfigured limits can cause silent request drops or node out-of-memory errors that inflate apparent throughput, so those figures should be read as ceilings rather than production targets.
Why it matters
The study makes concrete a point serving engineers have argued for some time: a single accelerator benchmark number says little unless you know the input and output shape of the traffic. Cloud providers are now publishing workload-specific guidance because customers choose between model sizes on unit economics. Autoscaling on latency rather than on resource utilisation is becoming the recommended pattern for inference services. The emphasis on silent failure at high concurrency is a reminder that benchmark ceilings and safe operating points are different things.
The study makes concrete a point serving engineers have argued for some time: a single accelerator benchmark number says little unless you know the input and output shape of the traffic.
- Gemma 3 12B, generation8.19x
- Gemma 3 27B, generation4.12x
- Gemma 3 12B, classification6.37x
- Gemma 3 27B, classification6.04x
Figures: Google Cloud benchmark; baseline is Gemma 3 12B at 16 users
What you can learn from this
Prefill and decode are two different computations. Prefill processes every input token in parallel to build the key-value cache, which is a large matrix multiplication that keeps compute units busy. Decode then produces one output token at a time, each step reading the whole model and the cache, which is bounded by memory bandwidth. A classification request is almost all prefill and a generation request is mostly decode, which is why the same chip and model behave so differently across the two. Time to first token is dominated by prefill and tokens per second thereafter by decode, so the two phases even show up as separate metrics.
Continuous batching is what lets many users share one replica. A serving engine such as vLLM groups requests that are at different stages and runs them together on each step, adding new prefill work and dropping completed sequences without waiting for a whole batch to finish. Throughput rises with concurrency until memory for the key-value cache or compute is saturated. The 27B model saturates earlier on generation because its larger weights and cache leave less room for parallel sequences.
Padding and bucketing shape how well hardware is used. Accelerators like TPUs compile kernels for fixed tensor shapes, so a serving engine rounds each batch up to the nearest supported size and fills the gap with zeros. If the bucket sizes are far apart, much of the work is wasted on padding, which is why the post exposes a padding-gap setting. Understanding this explains why throughput can grow in steps rather than smoothly as users are added.
Latency-based autoscaling matches what users experience. CPU and memory utilisation are poor scaling signals for inference because an accelerator can be fully allocated while still serving acceptably, or lightly loaded while queues build. End-to-end latency captures queueing, prefill and decode time together. Scaling on that signal adds replicas when responsiveness degrades rather than when a proxy metric crosses a threshold. Queue depth is a useful secondary signal for the same reason.
Silent failures corrupt benchmarks. If a server drops requests or a node runs out of memory under load, the requests that do complete may finish faster, so a naive throughput calculation improves while the service is actually failing. Sound benchmarks count errors and check that every request returned a full response. This is a general rule for any load test, not only for language models. Report the error rate next to every throughput figure.
We teach this
How to use this in practice
Classify your own prompts by shape. Take a sample of 50 requests from any application you have built or use, and record input tokens and output tokens for each using a tokenizer library such as tiktoken or the Transformers tokenizer. Done is a scatter plot with two visible clusters, or a note that your traffic is uniform, which tells you which regime the benchmark applies to. Keep the sample file; you will reuse it as the input for the load test in the next step.
Run a local load test that counts errors. Start a small model with vLLM or Ollama, use a tool such as
heyork6to send 16, 32 and 64 concurrent requests, and log throughput, p95 latency and the count of non-200 responses at each level. Done is a table where the error column is zero at every concurrency you plan to operate at, and a note of the level where it first stopped being zero. Watch memory on the server while the test runs so you can see whether a failure came from the queue or from the cache filling up.Set an explicit concurrency cap on a model server. Find the maximum-sequences setting for your engine, set it deliberately rather than leaving the default, and confirm that excess requests queue or receive a clear rejection instead of failing silently. Done is a test where sending one request above the cap produces a visible, logged outcome. Record the cap in your deployment notes so the next person knows it was chosen, not inherited.
Draw the request lifecycle for one inference call. Sketch the path from client through load balancer, queue, prefill, decode and response, marking where time is spent for a short-output and a long-output request. Done is a diagram you could use to explain why two requests of equal total tokens can have very different costs. Add the point where the cap from the previous step rejects a request, so the diagram reflects your real configuration.
Sources
- Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation — Google Cloud Blog
Our reporting is an original summary; full coverage is at the links above.
Don't just read about it — build it.
Square 1 teaches the skills behind the headlines, with every line of your work graded by AI. Find your starting point in 3 minutes.
Get your free skill report