Transformers MoE Streaming: 180B on 32GB, 13.9 tok/s
A Hugging Face blog post published on November 18, 2026 shows Transformers running a roughly 180B parameter MoE model on a Mac with 32 GB of unified memory, at 13.9 tokens per second. The trick is not compression alone. The expert weights of Qwen3.8-Flash-Next stay on the SSD and are streamed into a bounded device cache only when the router selects them, so memory holds a working set instead of the whole model. On the same machine and prompts, the dedicated engine Strata decoded at 14.8 tok/s, which puts the general purpose library about 6 percent behind on short decoding, and slightly ahead on prefill, 228 tok/s against 220 tok/s on a 4K prompt.
That comparison is the real news. Strata was built for exactly this job and showed the laptop result first. Transformers is the library most teams already load models with, and its standard offload path was nowhere near this class of result. Closing to within 6 percent inside the mainstream library changes who can experiment with very large MoE models on ordinary hardware, even though, as the author is quick to say, the speed is still not enough for his own coding agent work.
Measured Results on the 32 GB Mac
The final numbers come after a round of optimization that mattered more than the streaming itself. The first full version ran at about 10 tok/s for decoding, with prefill at 68 tok/s against Strata at 220 tok/s. Forcing GPU work to commit after each block added about 10 percent, because PyTorch on MPS otherwise holds small kernels until the next synchronization and leaves the GPU idle while Python works. Custom Metal kernels covered Gated DeltaNet and the hyper-connections, which had no fast path on macOS. Switching to Strata own GGUF weights cut dense weight traffic from 5.4 GB to 3.9 GB per token, and bigger prefill chunks plus reading the next layer experts during current layer compute lifted prefill from 68 to about 228 tok/s.
After a 6K document, decoding held at 12.4 tok/s against Strata at 13.7 tok/s, a gap of about 10 percent. Cache policy turned out not to be the lever the author expected. No policy beat plain LRU by much, and Strata does not hide cache misses either. What helped was using otherwise idle SSD time to read one guessed expert per layer, so a likely miss is already in flight before the router confirms it.
The baseline puts the gain in perspective, with one caveat about the setup. In a separate comparison run on Qwen3.6-35B-A3B with a single A10G GPU, Transformers with Accelerate offloading managed 0.16 tok/s, while Strata reached 82 to 88 tok/s and llama.cpp reached 63 to 64 tok/s with its fit option. Those figures are from a different model and GPU than the Mac run, so they do not belong on the same chart. They do show why the old offload path was not a contender at all, which is the gap the new implementation was built to close.
How the Slot Pool Cache Works
The first prototype kept experts on disk behind an LRU cache and ran them one at a time in a Python loop. Even with every expert already cached it reached only 10.1 tok/s, against about 32 tok/s for vanilla Transformers on a model that fits in memory. The I/O was fine. The loop was the problem, because vanilla Transformers runs all experts of a layer in a single grouped_mm kernel call.
The fix, and the central idea of the project, was to turn the cache into a slot pool. One preallocated device tensor is shared by all layers, and a per layer table maps expert IDs to slots. Remapping the router chosen IDs to slot numbers makes the pool look like a regular expert bank, so grouped_mm runs on it unchanged and no custom expert matmul is needed. On the small OLMoE-1B-7B test model, that change alone moved decoding from 10.1 to 34.6 tok/s, close to the 38.1 tok/s of the vanilla in memory run.
In the shipped form, experts mapped to disk in device_map take the streaming path instead of Accelerate offload. The loader skips their tensors, MoE layers switch to the offloaded implementation, and misses are read by 8 I/O threads, one contiguous read per expert. The routers of the next two layers run early on the current hidden states to guess which experts to prefetch. The approach works for MoE models with the standard experts layout, according to the post.
Fitting 121 GB of Experts Into a 32 GB Machine
Qwen3.8-Flash-Next has 48 layers with 512 experts each, 10 active per token, and the experts alone take 121 GB in FP8. Streaming alone does not solve the model, because a separate 51 GB n-gram embedding table also has to live somewhere. The setup therefore combines three moves. The experts are quantized to 3 bits, which brings them to 49 GB on disk with a 12 GiB device cache. The dense layers run in 8 bits at about 5.4 GB, at a reported cost of 0.2 percent perplexity against BF16. The n-gram table stays on disk and its rows are read when needed.
The conversion takes about 5 hours. After loading, memory use sits at about 18 GiB, and a 6K prompt peaks at 25.4 GiB. The author calls that roughly the limit of the machine, noting that the browser has to be closed or macOS starts swapping. The model specific pieces, the n-gram table handling and the Metal kernels, live in a small companion package on top of the Transformers pull request, and the kernels are loaded from the Hub with no C++ extension in the project.
To try it before it merges, the post installs Transformers from the pull request and loads the converted model like this:
pip install "git+https://github.com/huggingface/transformers.git@refs/pull/50717/head"
model = AutoModelForCausalLM.from_pretrained(
f"{path}/dense",
dtype=torch.bfloat16,
device_map={"": "mps", "experts.*": "disk"},
quantization_config=MetalConfig(bits=8),
experts_offload={
"cache_bytes": 12 << 30,
"quantized_experts": f"{path}/experts-q3",
},
)
That code is a snapshot of a moving target. The implementation is huggingface/transformers pull request 50717, surfaced as the experts_offload option of from_pretrained, and the API form is still being discussed with the maintainers. The post expects quantized experts to move to the per layer quantization plan of a separate pull request rather than keeping their own option, so scripts written against this shape should expect to change.
How the Other Engines Offload
The post surveys the alternatives before committing, and the differences explain why streaming is a distinct approach rather than a tuning flag. llama.cpp memory maps the GGUF file and can compute experts on the CPU, so only activations cross the bus and weights never move. That is hard to beat when the experts fit in RAM. When they do not, the page cache thrashes, and a recently landed GPU expert cache fragments the graph and loses about 10 percent even when everything fits.
vLLM offloads a set number of gigabytes to pinned RAM and reads weights in place, but nothing is cached, so a hot expert crosses PCIe again on every token, and there is no disk tier. Restricting offload to expert parameters doubled its speed in the comparison run, and FP8 helped further, but the design targets large batches that touch almost every expert anyway. That is the opposite of a single user on a laptop, where routing is sparse and a bounded cache captures most of the value. The author also tried vllm-metal, which recently gained expert offloading with a bounded cache, but it cannot run this model in 32 GB because its n-gram tables stay in memory at about 32 GB in the Hub checkpoints before dense layers and cache.
The old Transformers and Accelerate path offloads one module at a time, copying weights to the GPU before each forward pass. For an MoE layer, the module is the whole expert bank, so every token copies every expert of every offloaded layer to use 8 or 10 of them. There is no cache and no notion of routing, and on a Mac, where CPU and GPU share the same memory, offloading to the CPU frees nothing. Streaming only the routed experts through a bounded cache is a direct answer to each of those failures.
Limits and Unconfirmed Points
- Not merged yet: pull request 50717 is open and its API is under discussion, so neither the option name nor the loading flow is final.
- Too slow for the author own agent work: the conclusion states plainly that about 14 tok/s is still too slow for how he uses coding agents, and his dual RTX 4090 desktop keeps doing the real work.
- The remaining gap is structural: most of what is left is host round trips of about 10 milliseconds per token, and PyTorch does not expose control over the per layer synchronization that would be needed to close it.
- Single machine, single author: every figure in the post was measured by the author on one Mac M5. No independent reproduction has been published yet.
- Conversion cost and memory ceiling: the roughly 5 hour conversion and the 25.4 GiB peak on a 6K prompt leave little headroom on a 32 GB machine for anything else running alongside.
Read with those limits, the result still marks a shift in what decides the largest model a machine can run. For MoE models, where experts are more than 90 percent of the weights, the ceiling is moving from memory capacity to storage bandwidth and cache design. A 180B class model decoding at interactive speed on a 32 GB laptop, inside the library teams already use, widens the local model shortlist by a full tier. The author own verdict keeps the claim honest, though. Interactive is not the same as agent ready, and until the synchronization cost comes down, streamed experts are a way to run bigger models locally for reading, drafting and evaluation, not yet a replacement for a properly provisioned workstation.
FAQ
What did the Transformers expert streaming experiment achieve?
On a Mac with an M5 chip and 32 GB of unified memory, Transformers ran Qwen3.8-Flash-Next at 13.9 tokens per second for short decoding and 228 tokens per second for prefill on a 4K prompt. The dedicated Strata engine reached 14.8 tokens per second on the same decode test, so the general purpose library finished about 6 percent behind.
How does the slot pool expert cache work?
All layers share one preallocated device tensor that acts as an expert cache, capped at 12 GiB in this setup. A per layer table maps expert IDs to cache slots, and the router output is remapped to those slots. The pooled cache then looks like a normal expert bank, so the existing grouped_mm kernel can run on it without a custom expert matmul.
How can a 180B model fit on a 32 GB machine?
It does not fit in memory. The experts were quantized to 3 bits, which brought them to 49 GB on disk, and only routed experts are read into the bounded cache. The dense layers run in 8 bits at about 5.4 GB, and the 51 GB n-gram embedding table stays on disk with rows read on demand. Memory use after loading was about 18 GiB, peaking at 25.4 GiB on a 6K prompt.
How do I try experts_offload today?
The implementation is in huggingface/transformers pull request 50717, which is not merged yet. The source post installs Transformers from that pull request branch, then loads the model with experts mapped to disk in device_map and an experts_offload option that sets cache_bytes and points to the quantized experts. The API shape is still under discussion with the maintainers, so it may change before release.
Is 13.9 tok/s fast enough for coding agents?
The author says no. The post concludes that the result is close to Strata and still too slow for how he uses coding agents, so his dual RTX 4090 desktop keeps doing the real work. The remaining gap is attributed to host and GPU synchronization round trips of about 10 milliseconds per token, which PyTorch does not currently expose control over.
Related Reading
- llama.cpp v1.0.0 Release: API Stability and What Changes, the engine this post benchmarks as the strong baseline when experts fit in RAM
- Alpie Seed 9B Release: Offline Coding, 61.2 SWE-bench, a smaller model built for fully offline local coding