🔍 Read the full analysis: A Closer Look At 180B-Parameter MoE Inference On 32GB RAM on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A Hugging Face developer proposed an experimental Transformers option that streams selected mixture-of-experts weights from storage into a bounded device cache. In a reported OLMoE test, it generated 34.6 tokens per second versus 38.1 for vanilla Transformers; results for a 180B model on a 32 GB Mac have not been reported.
A Hugging Face developer has proposed an experimental SSD-backed expert-streaming option for Transformers, aiming to run large mixture-of-experts models without keeping all their weights in fast memory, as described in the original analysis. The developer reports 34.6 tokens per second on an OLMoE test, close to 38.1 tokens per second for vanilla Transformers, but provides no completed benchmark for the motivating 180B model on a 32 GB Mac.
The approach targets a memory bottleneck in mixture-of-experts (MoE) models. These models route each token through only some of their experts, rather than using every expert for every token. The developer says experts account for more than 90% of the model weights, making it possible in principle to load only the selected experts as needed while leaving others on storage.
The developer first tested the idea on OLMoE-1B-7B. A basic version ran experts one at a time in a Python loop and reached 10.1 tokens per second even with experts already cached, compared with about 32 for vanilla Transformers. The revised design uses a shared, preallocated device tensor and a table mapping expert IDs to cache slots. That lets the existing grouped matrix multiplication kernel process selected experts together; the reported result was 34.6 tokens per second, against 38.1 for the standard path.
The work is described as submitted to Transformers through pull request #50717. The post also points to a Hugging Face Space named tarekziade/moe-expert-streaming, where the project code is available. The developer says the work involved Transformers, a model-specific package and Metal kernels, with AI used in coding; they describe their role as choosing experiments, reviewing profiles and deciding which implementation changes to keep.
Streaming MoE Experts With Limited Memory
An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.
How the cache works
The router selects experts for each token; the experiment fetches those experts into a shared cache.
Choose a subset of experts for the token.
Stream them from storage into available device cache slots.
Use Transformers’ grouped matrix multiplication kernel.
What the test shows
On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.
- Python loop ran experts one at a time.
- Reached 10.1 tokens per second, even with experts cached.
- Preallocated tensor mapped expert IDs to cache slots.
- Reached 34.6 tokens per second versus 38.1 baseline.
The larger device test is open
The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.
Promising engineering
The cache design brought the reported small-model test close to vanilla Transformers speed.
SSD performance
Storage traffic and cache misses could change speed when experts must be fetched repeatedly.
Measure the target setup
Report speed and memory use on the larger model, with storage, cache settings and workload specified.
Making Larger MoE Models More Accessible
If the approach performs well when experts must actually be fetched from storage, it could let developers run some larger MoE models on hardware with limited memory. That may be relevant to local inference on laptops and single-GPU systems, where the full model weights can exceed available memory even though each token uses only a subset of the experts.
The key trade-off is between memory capacity and data-movement speed. Streaming can reduce the amount of weight data that must remain in device memory, but storage reads, transfers and cache misses may slow generation. The OLMoE result supports a narrower point: the revised cache design avoided much of the slowdown seen in the initial Python-loop implementation on that test. It does not establish how quickly a much larger model runs when experts have to be loaded from an SSD.
The distinction matters for people evaluating local model use. A promising result on a smaller test model can show that an implementation path is workable without confirming that it meets practical speed or memory needs on the larger target. Until that test is reported, the proposed 180B use case remains a goal, not a demonstrated result.
Products Worth Considering

Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black

Samsung 32 Odyssey G5 QHD Gaming Monitor

LG 27GR83Q-B 27-inch Ultragear QHD IPS Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Expert Streaming Is Designed
The developer’s design keeps a bounded group of experts in a shared device cache and brings in routed experts as needed. This differs from the tested Python-loop version, which processed experts one at a time and ran substantially slower than the baseline on OLMoE. The revised design retains grouped expert computation, which the developer credits for bringing the reported speed close to vanilla Transformers.
The post also compares other approaches using Qwen3.6-35B-A3B on an A10G with 24 GB of GPU memory and ample system RAM. In that specific setup, it reports Strata at 82–88 tokens per second with Q8_0 and llama.cpp at 63–64 with Q8_0 and its fit option. The author describes those results as setup-specific; they do not measure the new Transformers option or predict laptop performance.
In the author’s account, llama.cpp can place experts on the CPU when they fit in RAM, while performance may suffer when the system’s page cache is under pressure. The post characterizes vLLM’s CPU offload as reading offloaded weights over PCIe without a hot-expert cache, and Transformers with Accelerate as moving an entire layer’s expert bank before processing it. These are the author’s descriptions of the compared systems, not results from a matched benchmark of all options.
“The revised cache design reached 34.6 tokens per second, versus 38.1 for vanilla Transformers.”
— The developer, reporting the OLMoE test
The 180B Mac Benchmark Is Missing
The source does not report a completed run of the experimental Transformers implementation on the 180B model and 32 GB Mac that motivate the work. It also does not provide the OLMoE test’s storage type, cache size, prompt conditions or repeated-run results, limiting what can be concluded about performance across workloads.
The post says pull request #50717 was submitted, but does not establish whether it was accepted, merged or included in a Transformers release. It is also unclear how the cache performs when routing changes the required expert set quickly, or how the new option compares with Strata on the same model and hardware. The reported OLMoE speed is therefore evidence for one test, not a general performance guarantee.
Pull Request and Matched Tests
The next concrete development is the status of Transformers pull request #50717 and whether the experimental option becomes available in a released version. The linked Hugging Face Space offers a way to inspect the project code, according to the post.
To assess the central use case, future results would need to report generation speed and memory use for the intended 180B MoE model on a 32 GB Mac, including storage device, cache settings and workload. Repeated runs and a comparison with Strata on matching hardware and conditions would help show whether the approach remains fast when experts must be streamed from storage rather than already being cached.
Key Questions
What is the experimental change?
It is an SSD-backed expert-streaming option proposed for Transformers. The design aims to load routed MoE experts into a bounded device cache instead of keeping all expert weights in fast memory.
Was the 180B model tested on a 32 GB Mac?
The source presents that setup as a motivating target but reports no completed benchmark for it.
How fast was the reported OLMoE test?
The developer reports 34.6 tokens per second for the revised design and 38.1 for vanilla Transformers. Those figures are from the OLMoE test and should not be treated as results for the larger model.
Is the feature available in Transformers now?
The post says the change was submitted as pull request #50717. It does not say whether the request was accepted, merged or released.
What would confirm the larger-model claim?
A reported test on the 180B model and 32 GB Mac should identify the storage device, cache settings, workload, memory use and generation speed. A comparison under the same conditions would help readers assess the result.
Primary source: Hugging Face · via ThorstenMeyerAI.com
