Nvidia unveils RTX Spark compact PCs and a PAIR local inference router at IFA 2026

Model size in memory is parameters times bytes per weight. A model with 70 billion parameters stored at 16-bit precision needs roughly 140GB just for its weights, while 4-bit quantisation brings that to about 35GB. The GPU must also hold the key-value cache, which grows with context length and batch size. This arithmetic is why 24GB and 128GB appear as thresholds in vendor material: they mark which model classes fit without spilling to slower storage.

Unified memory removes the copy step between CPU and GPU. In a conventional PC the graphics card has its own memory, and data must cross the PCIe bus before the GPU can use it, which limits model size to the card's VRAM. Unified memory lets the CPU and GPU address one pool, so a model can be as large as the whole system's memory allows. The trade-off is that shared memory is usually slower per byte than dedicated graphics memory, so bandwidth rather than capacity becomes the ceiling on tokens per second.

Inference throughput depends on memory bandwidth more than compute. Generating each token requires reading every weight of the model once, so a chip that streams memory faster produces tokens faster, even with the same arithmetic capacity. Speedups such as those claimed for llama.cpp and vLLM typically come from fusing kernels, better batching and cutting redundant memory traffic. Understanding this explains why software updates alone can lift performance on cards that have not changed.

A local inference router is a load balancer for models. A tool like PAIR sits in front of several machines that each run a model server, accepting requests through a single endpoint and forwarding them to whichever node has capacity. This is the same pattern web applications use with reverse proxies, applied to model servers such as Ollama. The router must also know which node holds which model, since loading weights takes seconds to minutes and cannot be shuffled per request.

Agent runtimes need a model, a tool interface and a loop. Applications such as Hermes Agent or OpenClaw wrap a language model in a control loop that reads a goal, chooses a tool, observes the result and repeats. Running that loop locally means the model, its tools and any files it touches all stay on the machine, which is the privacy argument for local agents. It also means the machine must sustain many sequential model calls, which is why vendors emphasise throughput rather than single-response latency.