Nvidia claims Vera Rubin NVL72 delivers up to 30x more work per watt on agentic AI workloads
Vendor benchmarks built on multi-step agent sessions show why tokens per watt, not peak FLOPS, is becoming the number that matters.
Nvidia published performance figures on 24 August 2026 claiming that its Vera Rubin NVL72 rack-scale system delivers up to 30 times more throughput per megawatt than the previous-generation GB300 NVL72 on agentic AI workloads. The company also said token costs on the new system are up to 35 times lower than on GB300 NVL72, and noted that GB300 NVL72 itself delivers up to 15 times better throughput per megawatt than the older Hopper architecture.
The results were measured with the SemiAnalysis AgentX workload, which is built from real-world agentic coding sessions rather than single-turn prompts. Models tested included Kimi K3, MiniMax M3, GLM5.3, Qwen3.5 and DeepSeek V4 Pro. Nvidia stated that the results are currently pending review by SemiAnalysis.
Nvidia frames the announcement around an observation it attributes to OpenRouter data: agentic AI workloads consume 15 times more tokens than a simple chat request. Its illustration is an agent researching a company for an investment decision. The agent queries financial databases, searches news and filings, invokes a sub-agent to run peer comparisons and valuation models, then synthesises everything into a recommendation. Each step generates tokens, and the total grows quickly.
Vera Rubin NVL72 is described as a seven-chip architecture. The post lists the Vera CPU, Groq 3 LPU, NVLink 6 Switch, BlueField-4 DPU, Spectrum-6 SPX and ConnectX-9 SuperNIC alongside the Rubin GPU that gives the system its name. Nvidia said Vera Rubin is in full production and scaling across its ecosystem. As with any vendor-published benchmark, the figures reflect a workload and configuration chosen by Nvidia and have not yet been independently confirmed.
Why it matters
Power, not silicon, is now the binding constraint on AI infrastructure, so the industry has shifted from quoting raw compute to quoting useful work per watt. Nvidia's decision to benchmark on multi-step agent sessions rather than chat prompts reflects where inference demand is heading. The inclusion of a dedicated language processing unit alongside GPUs also signals that inference is becoming heterogeneous, with different chips handling different phases of a request. Whether the headline multiples survive independent review will shape how buyers plan the next round of data centre capacity.
Power, not silicon, is now the binding constraint on AI infrastructure, so the industry has shifted from quoting raw compute to quoting useful work per watt.
- Throughput per megawatt, Vera Rubin NVL72 vs GB300 NVL7230x
- Lower token cost, Vera Rubin NVL72 vs GB300 NVL7235x
- Throughput per megawatt, GB300 NVL72 vs Hopper15x
- Agentic tokens vs a simple chat request (OpenRouter data)15x
Figures: Figures published by Nvidia; agentic results described as pending SemiAnalysis review
What you can learn from this
- Throughput per watt is the metric that matters. A data centre is typically permitted for a fixed amount of electrical power, so every watt spent on one request is a watt unavailable to another. Measuring tokens per second per megawatt captures how much useful output a facility can deliver regardless of how many chips are inside it. Vendors emphasise this figure because it converts directly into revenue capacity for an operator with a fixed power budget. It also explains why efficiency gains can matter more than peak performance.
- Prefill and decode behave differently. Serving a language model has two phases: prefill, where the whole prompt is processed in parallel and the work is compute-bound, and decode, where tokens are produced one at a time and the work is limited by how fast weights and cached state can be read from memory. Agentic workloads are decode-heavy because they produce many tokens across many turns. A chip optimised for decode, such as a processor with large on-chip memory, can outperform a general GPU on that phase even if it is weaker on prefill. That is the logic behind pairing different processors in one rack.
- Why agents multiply token counts. A chat request is one prompt and one answer, but an agent loops: it plans, calls a tool, reads the result, reasons again and often delegates to sub-agents. Each loop re-reads a growing context and produces new tokens, so the total scales with the number of steps and the length of accumulated history. This is why the same model can cost an order of magnitude more per task in agentic mode. Caching the key-value state of earlier turns reduces the repeated prefill cost but does not remove the decode cost.
- Rack-scale systems and the scale-up domain. NVL72 refers to 72 GPUs connected by a high-bandwidth fabric so they behave like one large accelerator. Large models, especially mixture-of-experts designs, spread their weights across many chips, and each token may need data from several of them. A fast, low-latency interconnect keeps that exchange from becoming the bottleneck. Networking chips and switches are therefore as much a part of the performance story as the GPUs.
- How to read a vendor benchmark. The numbers here come from a workload chosen by the vendor, at a batch size and precision the vendor selected, and are explicitly awaiting third-party review. None of that makes them wrong, but it means they describe a best case for a particular scenario. Comparing generations of the same vendor's hardware is more reliable than comparing across vendors, because the methodology is at least consistent. A useful habit is to ask what changes if the workload were prefill-heavy, low-batch or latency-sensitive.
We teach this
How to use this in practice
- Measure your own tokens per watt. Run a small open model locally with llama.cpp or Ollama, generate a 500-token response, and record the reported tokens per second. In a second terminal, sample power draw with
nvidia-smi --query-gpu=power.draw --format=csv -l 1on an Nvidia GPU, or your platform's equivalent. Divide throughput by average power. Done looks like a single number in tokens per second per watt, plus the same test repeated with a much longer prompt so you can see the prefill cost. Repeat with a quantised version of the same model to see how precision changes both throughput and power. - Count tokens in an agent loop. Take any agent framework you use, or a 40-line script that calls a model in a loop with one tool, and log the prompt and completion token counts from each API response. Run the same task once as a single chat prompt and once as a multi-step agent. Done looks like a ratio of agent tokens to chat tokens for your task, which you can set beside the 15x figure Nvidia cites. Note how much of the agent total is prompt tokens re-sent on each turn.
- Draw the two phases. Sketch a request timeline showing prefill as one wide block and decode as a series of thin blocks, then annotate which is compute-bound and which is memory-bound. Add a second row showing what a cached second turn looks like. Done looks like a diagram you can use to explain why long agent sessions are dominated by decode.
- Build a benchmark checklist. Write down six questions to ask of any hardware performance claim: which workload, which models, what batch size, what precision, whether results are independently reviewed, and what the comparison baseline is. Apply it to this announcement and to one other vendor's recent claim. Done looks like a filled-in table with both entries, including blanks where the vendor did not say. Keep the table; it becomes a template for every future launch.
Sources
- Up to 30x More Work Per Watt: NVIDIA Vera Rubin NVL72 Sets a New Efficiency Standard for AI Agents — NVIDIA Blog
Our reporting is an original summary; full coverage is at the links above.
Don't just read about it — build it.
Square 1 teaches the skills behind the headlines, with every line of your work graded by AI. Find your starting point in 3 minutes.
Get your free skill report