AMD acquires inference startup Taalas, which hardcodes model weights into silicon
Fixing a model's weights into a chip's metal layers trades flexibility for speed, and explains where inference time is really spent.
AMD has acquired Taalas, an AI inference startup founded by former Tenstorrent engineers, for an undisclosed sum, The Next Platform reported on 7 August 2026. AMD said it plans to fold the technology into its accelerator roadmap and to build system-level products alongside its Instinct GPUs.
Taalas's approach, which it calls Model Specific Architectures, stores a model's weights directly in read-only memory circuits on the chip and pairs them with large SRAM blocks that serve as an on-chip key-value cache. Each model needs its own variant of the company's HC1 chiplet: the SRAM and surrounding logic are unchanged, but two metal layers that encode the weights are altered per model. The article puts the cost of customising an HC chip at roughly one-hundredth of the cost of training a new large language model.
The current HC1 generation holds models of up to 8 billion parameters, and the planned HC2 was slated to reach 20 billion. The article notes that, in principle, a few tens of interlinked chips could hold a trillion-parameter model. Taalas has demonstrated very low latency and lower cost per token than Nvidia's Blackwell B200 GPUs, according to the report.
The piece frames the purchase as part of a move toward disaggregated inference, where prefill and decode run on different hardware. It cites Nvidia analysis that GPU-only systems reach about 100 tokens per second per user at a medium tier and up to 400 at a premium tier before returns fall away, while pairing GPUs with a decode accelerator pushed past 1,000 tokens per second per user.
Why it matters
Inference is now where most AI compute is consumed, and the decode phase is limited by memory bandwidth rather than arithmetic. That has created room for hardware that gives up generality to keep weights physically next to the compute. AMD's purchase signals that major GPU vendors expect to sell mixed systems with different silicon for different phases of a request.
Inference is now where most AI compute is consumed, and the decode phase is limited by memory bandwidth rather than arithmetic.
General-purpose GPU
Weights stream from HBM on every decode step; runs any model; per-user token rate plateaus as memory bandwidth saturates.
Taalas HC chiplet
Weights fixed in two metal layers, KV cache in on-chip SRAM; one model per variant; low latency and lower cost per token, per the report.
What you can learn from this
- Prefill and decode stress hardware in opposite ways. Prefill processes every token of the prompt at once, which is a large parallel matrix operation and keeps a GPU's arithmetic units busy. Decode produces one token at a time, and for each token the entire set of model weights must be read from memory again. That is why per-user token rates plateau on GPUs: the chip is waiting on memory, not on maths. Batching many users together raises total throughput but does nothing for the latency any single user experiences.
- The memory hierarchy sets the speed limit. Weights normally live in high-bandwidth memory beside the GPU die, which is large but comparatively slow to stream from; SRAM on the chip itself is far faster but tiny; read-only circuits are faster still and take no power to hold their contents. Placing weights in the fastest tier removes the streaming bottleneck, at the cost of fixing them permanently. Every accelerator design is a bet on where along this hierarchy to put the bytes that matter.
- The key-value cache is the other big memory consumer. As a model generates text, it stores intermediate attention vectors for every previous token so it does not have to recompute them, and that cache grows with context length. Keeping it in on-chip SRAM cuts the latency of each decode step because the cache is read on every token. This is also why long-context requests cost more to serve than short ones. Serving systems therefore cap context length or evict old cache entries to keep memory within budget.
- Fixed-function silicon trades flexibility for efficiency. A general-purpose chip can run any model but pays for that flexibility in bandwidth and power; a chip built for one model runs only that model but does so with less waste. Changing just two metal layers is much cheaper than a full new chip design, which makes the trade-off viable when a model is stable and served at high volume. The bet only pays off if the model does not need to change every few months. It also shifts risk onto the chip maker, who must predict which models will still be in demand when the silicon ships.
- Disaggregated serving improves utilisation by matching work to hardware. Running prefill on a pool of compute-heavy devices and decode on a pool of bandwidth-optimised devices lets each pool stay busy with the work it is good at. The cost is a hand-off of the key-value cache between pools, so interconnect speed becomes part of the design. Systems built this way are more complex to schedule but cheaper per token at scale.
We teach this
How to use this in practice
- Measure prefill and decode on your own machine. Run a small open model locally with a tool such as llama.cpp or Ollama in verbose mode, give it a long prompt, and note the reported prompt-evaluation rate and the generation rate in tokens per second. Done looks like two numbers written down, with the generation rate visibly lower than the prompt rate. Repeat with a much shorter prompt and confirm the generation rate barely changes, which shows decode cost is paid per token rather than per prompt. Record the model name and quantisation level next to the numbers so the result can be reproduced.
- Do the bandwidth arithmetic. Take the model's size in bytes at the precision you ran, divide by your GPU's or CPU's memory bandwidth in bytes per second, and invert to get a theoretical maximum tokens per second for a single user. Compare it with the measured decode rate from the first item, keeping the units consistent, bytes and bytes per second, or the ratio will be off by a large factor. Done looks like a one-line calculation and a short note on how close reality came to the ceiling. If the gap is large, record whether the run was limited by memory or by something else, such as offloading layers to the CPU.
- Sketch your machine's memory hierarchy. Draw registers, on-chip cache or SRAM, main or HBM memory, and storage as stacked boxes, and label each with its approximate size and bandwidth from your hardware's specification sheet. Done looks like a labelled diagram that makes the size-versus-speed trade-off visible at a glance. Mark where a model of the size you ran would sit and whether it fits in any tier above main memory. Use the specification sheet's figures rather than guesses, and note the source.
- Compare a quantised and an unquantised model. Run the same prompt through the same model at two precisions, for example 16-bit and 4-bit, and record decode speed and any change in answer quality you notice. Done looks like a two-row table showing that halving the bytes per weight raised the token rate, which is the same lever hardware designers are pulling. Write one sentence on whether the quality change would be acceptable for a task you actually have.
Sources
- With Taalas, AMD Can Bake AI Inference Directly Into Its Chippery — The Next Platform
Our reporting is an original summary; full coverage is at the links above.
Don't just read about it — build it.
Square 1 teaches the skills behind the headlines, with every line of your work graded by AI. Find your starting point in 3 minutes.
Get your free skill report