Nvidia adds NVHBM custom memory to NVLink Fusion, with Amazon's Trainium4 first in line
Moving the memory controller off the accelerator die illustrates how bandwidth, power and silicon area trade against each other in AI chips.
Nvidia said on 26 August that it is adding a custom high-bandwidth memory technology, NVHBM, to its NVLink Fusion programme, with Amazon's Annapurna Labs named as the first collaborator. According to the company's blog post, NVHBM moves Nvidia's custom memory controller off the accelerator die and into the base die of the HBM stack itself.
Nvidia said the design is built on technology it plans to use in its own future GPUs and that leading memory partners will validate and offer it. Compared with standard HBM4E, the company claims up to 30 percent higher memory bandwidth, 15 percent lower HBM power consumption and up to 25 percent more free area on the accelerator's compute die, because the controller no longer has to sit there.
Annapurna Labs, Amazon's chip design unit, will adopt NVHBM in its next-generation Trainium accelerators, starting with Trainium4. Nafea Bshara, vice president of Annapurna Labs, described NVHBM in the post as a new architectural approach to improving high-bandwidth memory performance and efficiency.
NVLink Fusion is Nvidia's rack-scale platform for partners building their own XPUs and CPUs. It gives them access to NVLink chiplets, NVLink-C2C, NVLink switches and the MGX system and rack designs, alongside an ecosystem of CPU partners, ASIC designers and system manufacturers. NVHBM extends that catalogue from interconnect into memory. The post, written by Jesse Clayton, framed the move as part of designing compute, memory, storage, networking and software together as a single system for agent and trillion-parameter workloads. No pricing, volumes or shipping dates were given.
Why it matters
Memory bandwidth, not arithmetic, is the binding constraint for most inference workloads, so a bandwidth gain is worth more than a similar gain in raw compute. Supplying a memory controller to a rival accelerator shows Nvidia positioning its memory and interconnect stack as infrastructure other chips are built around. The HBM base die is turning from passive routing into a place where logic lives.
Memory bandwidth, not arithmetic, is the binding constraint for most inference workloads, so a bandwidth gain is worth more than a similar gain in raw compute.
Standard HBM4E
Controller sits on the accelerator compute die; the HBM base die mostly routes signals; a standard interface works with any compliant host.
NVHBM
Nvidia's custom controller moves into the HBM base die; claimed up to 30% more bandwidth, 15% lower HBM power and 25% freed compute-die area.
What you can learn from this
- HBM is a stack, and the base die is its foundation. High-bandwidth memory places several DRAM dies on top of one another, connected by vertical through-silicon vias, and sits beside the processor on an interposer. Under the DRAM sits a base die that handles signalling to the host. Historically that die did little more than route signals; putting a controller there turns it into a logic die that can schedule and manage the DRAM above it directly. Each stack is only a few millimetres across, so every function moved into it must be small and power-efficient.
- A memory controller is a scheduler, not a wire. The controller queues reads and writes, groups them to avoid costly row switches, handles refresh cycles and manages error correction. On a conventional design it lives on the accelerator, consuming die area and, critically, edge length, because the wide HBM interface needs to exit the chip near the stack. Freeing that area gives designers room for more compute or more interface lanes. The interface is wide because HBM uses thousands of relatively slow signals in parallel rather than a few fast ones, which is what makes edge length so precious.
- Data movement costs more energy than arithmetic. Moving a bit across a package consumes far more energy than a multiply-add, and that cost grows with distance and signalling complexity. Shortening the path between controller and DRAM, and simplifying the interface across the interposer, reduces energy per bit. That is where a claimed power reduction can come from without changing the DRAM cells themselves.
- Arithmetic intensity tells you which resource limits you. The roofline model compares a workload's operations per byte with a chip's compute and bandwidth limits. Generating one token in a large model reads the whole weight set for only a handful of operations per weight, which is very low intensity, so the achievable rate is roughly bandwidth divided by model size. That is why memory bandwidth improvements translate almost directly into tokens per second for inference. Training has far higher intensity because each weight is reused across a whole batch, so it tends to be compute-bound instead.
- Custom interfaces trade interoperability for performance. JEDEC standards such as HBM4E let any compliant memory work with any compliant processor. A custom base die with a proprietary controller can outperform the standard because it is co-designed with a specific host, but it ties the memory to that ecosystem. Buyers gain speed and give up some freedom to mix suppliers, which is a business decision as much as a technical one.
We teach this
How to use this in practice
- Work out the bandwidth ceiling for a model you use. Multiply the parameter count by bytes per parameter (2 for 16-bit weights, 1 for 8-bit), then divide the accelerator's published memory bandwidth by that number. Done looks like: a single figure, tokens per second for one sequence, that you can compare with what you actually observe. If observed throughput is far below it, the bottleneck is somewhere else. Repeat the calculation for a quantised version of the same model to see how bytes per parameter changes the ceiling.
- Measure achieved bandwidth on whatever GPU you have. In PyTorch, allocate a tensor of a few gigabytes on the device, time a
.clone()betweentorch.cuda.synchronize()calls, and divide bytes moved by seconds. Done looks like: achieved GB/s expressed as a percentage of the spec-sheet number, which tells you how much headroom simple copies leave on the table. Run the copy several times and keep the median, because the first iteration includes allocation overhead. - Build a two-column spec comparison. Take the public data sheets for two accelerators and fill a table with HBM capacity, HBM bandwidth, peak dense FP16 or FP8 throughput and board power. Add a computed row for bytes per FLOP. Done looks like: a table that makes obvious which chip is built for bandwidth-bound inference and which for compute-bound training. If a data sheet quotes sparse throughput, record the dense number instead, since most workloads do not benefit from structured sparsity.
- Draw the memory stack. Sketch the layers from top to bottom: DRAM dies, through-silicon vias, base die, interposer, package substrate, with the compute die beside it. Then draw a second version with the controller moved from the compute die into the base die and shade the freed area. Done looks like: two diagrams that explain at a glance what an NVHBM-style change alters and what it leaves untouched. Label the through-silicon vias and the interposer traces so the drawing shows where signals actually travel.
Sources
Our reporting is an original summary; full coverage is at the links above.
Don't just read about it — build it.
Square 1 teaches the skills behind the headlines, with every line of your work graded by AI. Find your starting point in 3 minutes.
Get your free skill report