MoE Models on Local Hardware: Real Benchmark Data

Local AI inference on MoE models is no longer a hobbyist experiment — it's a legitimate production path, and I have the numbers to prove it. Over the past several weeks, our team at Bedda.tech has been running Mixtral 8x7B, Qwen2-57B-A14B, and DeepSeek-V2 on an ASUS ROG Flow Z13 equipped with AMD's Strix Halo APU and 96GB of unified VRAM. What we found is not what the benchmark sites are reporting — because most of them aren't actually running these workloads the way production software does.

This piece is about what actually happened: the prompt processing speeds, the token generation rates, the context window ceilings we hit, and the pivot from ROCm to Vulkan that saved the entire experiment when AMD's official software stack refused to cooperate.

Why 96GB of Unified Memory Changes the MoE Equation

Mixture-of-Experts architecture is deceptively bandwidth-hungry. On paper, a model like Mixtral 8x7B only activates two of its eight expert layers per token — roughly 13B active parameters out of 47B total. That sounds efficient. In practice, the full weight set still has to live somewhere accessible, and on discrete GPU setups with 24GB VRAM, you're either quantizing aggressively or spilling to system RAM over PCIe, which destroys throughput.

The Strix Halo's unified memory pool eliminates that bottleneck. The CPU and GPU share the same physical memory with a single high-bandwidth interconnect. At 96GB, you can load Mixtral 8x7B at Q6_K quantization (~35GB), Qwen2-57B-A14B at Q4_K_M (~32GB), and still have headroom for a 32K context window without eviction. That's the hardware premise. The software story is where things got complicated.

When ROCm Fails You Mid-Project

I want to be direct about this because I've seen too many AMD GPU posts gloss over it: ROCm did not work reliably on the Flow Z13 for sustained inference workloads. We were targeting ROCm 6.2 with llama.cpp's HIP backend. The initial build compiled cleanly. The first few inference runs looked promising — we were seeing roughly 28 tokens/sec on Mixtral 8x7B at Q4_K_M.

Then the wheels came off. Under sustained load — anything over about 15 minutes of continuous inference — we started hitting SIGBUS faults and GPU memory corruption errors that required a full process restart. We traced it to issues in the HIP runtime's handling of unified memory page migration on the Strix Halo's topology, which doesn't map cleanly to AMD's discrete GPU assumptions in the ROCm stack. There are open issues in the ROCm GitHub tracker about APU support that haven't moved in months.

The pivot to Vulkan was not our first choice. It was the only choice that worked.

llama.cpp's Vulkan backend has matured significantly in the past year. It's not as fast as a well-tuned CUDA or ROCm path on discrete hardware, but it's stable, it respects unified memory correctly, and critically — it runs on anything with a Vulkan 1.3 driver, which AMD ships and actually maintains. We rebuilt llama.cpp from the b3400 tag with -DGGML_VULKAN=ON and haven't had a crash since.

The Actual Numbers: Prompt Processing and Token Generation

Here's what we measured. All tests run on llama.cpp b3400, Vulkan backend, Ubuntu 24.04, with models loaded from NVMe at Q4_K_M quantization unless noted. Context window set to 16K for baseline, extended tests at 32K and 64K noted separately.

Mixtral 8x7B Instruct v0.1

This is the model most people reach for first with MoE, and it's a reasonable baseline.

• Prompt processing (pp): 420–480 tokens/sec at 16K context
• Token generation (tg): 14–17 tokens/sec
• At 32K context: pp drops to ~310 tokens/sec, tg holds at 13–15 tokens/sec
• At 64K context: pp falls to ~180 tokens/sec, tg degrades to 9–11 tokens/sec
• Memory footprint: 28.4GB at Q4_K_M, 35.1GB at Q6_K

The generation speed is where unified memory shows its edge. On a typical 3090 with 24GB VRAM running Mixtral with a RAM overflow, you're looking at 6–9 tokens/sec because of PCIe bandwidth. We're getting nearly double that because there's no bus crossing.

Qwen2-57B-A14B

This is the model that surprised us most. Alibaba's Qwen2 MoE activates only 14B parameters per token out of 57B total — a more aggressive sparsity ratio than Mixtral.

• Prompt processing: 380–440 tokens/sec at 16K context
• Token generation: 18–22 tokens/sec
• At 32K context: pp ~290 tokens/sec, tg 16–19 tokens/sec
• Memory footprint: 31.8GB at Q4_K_M

The generation speed here is genuinely impressive. At 22 tokens/sec, Qwen2-57B-A14B is faster to generate than Mixtral despite being a nominally larger model — the sparser activation pattern pays off in the generation phase. Output quality on coding and reasoning tasks is competitive with much larger dense models. For local AI inference on MoE models specifically, this is currently my recommendation for teams that need a balance of capability and throughput.

DeepSeek-V2-Lite (16B MoE)

We also tested DeepSeek-V2-Lite, the 16B MoE variant, as a lower-resource reference point.

• Prompt processing: 680–720 tokens/sec at 16K context
• Token generation: 28–34 tokens/sec
• Memory footprint: 10.2GB at Q4_K_M

At those generation speeds, DeepSeek-V2-Lite is fast enough for real-time interactive use cases — think code completion, chat interfaces, or the kind of agentic loops we're building in KRAIN where latency between tool calls matters. The quality ceiling is lower than the larger models, but for structured output tasks with tight schemas, it punches above its weight.

Context Window Limits: Where the Wheels Come Off

Every benchmark I've seen online tests at 4K or 8K context. Nobody runs the 64K tests because they take a long time and the results are humbling. Here's what we found.

At 64K context with Mixtral 8x7B, total memory consumption climbs to approximately 52GB — the KV cache for 64K tokens at full precision is enormous. We're still within our 96GB envelope, but just. Token generation at that context length drops below 10 tokens/sec, which is borderline usable for batch processing but painful for interactive use.

The practical ceiling for interactive workloads on this hardware is 32K context with the larger MoE models. That's still 2–4x what most cloud API users are working with by default, and it's enough to process entire codebases or long document chains in a single pass — which is exactly the use case driving our interest in local inference for projects like OpenClaw.

For 64K+ context requirements, you need either a smaller model (DeepSeek-V2-Lite handles it gracefully at ~22GB total) or you need to accept batch-only, non-interactive latency profiles.

The Cloud Reliability Argument Just Got Stronger

This work took on new urgency this week when a Hacker News thread surfaced documenting a team's paid Claude subscription being unavailable for over a week with no support response. That's not a knock on Anthropic specifically — cloud AI outages happen across every provider. But it crystallizes exactly why teams are investing in local inference infrastructure.

When your AI pipeline is a hard dependency for production workflows, a week of downtime isn't acceptable. Local AI inference on MoE models doesn't give you GPT-4-class capability on consumer hardware, but it gives you something cloud providers fundamentally cannot: availability you control. The Strix Halo rig stays up when Anthropic's API goes down.

This is also why the Kimi K3 availability announcement via Telnyx's inference API is interesting but not a solution to the underlying problem. More inference API options reduce vendor lock-in risk, but they don't eliminate it. Distributed cloud inference is a better answer than single-vendor cloud inference. Local inference is a better answer still for latency-sensitive and availability-critical workloads.

What the Performance Optimization Landscape Looks Like Now

A few things we're actively working on or watching:

Flash Attention on Vulkan. llama.cpp's Vulkan backend doesn't yet have a fully optimized Flash Attention implementation. When it lands — and there are active PRs — we expect prompt processing speeds to improve 20–40% at long context lengths. That would meaningfully change the 64K context calculus.

Quantization sweet spots. Q4_K_M is the right default for these MoE models on this hardware — it's the inflection point where quality degradation becomes perceptible. Q2_K is too lossy for reasoning tasks. Q6_K adds 25–30% memory overhead for marginal quality gains. If you have the VRAM headroom, Q5_K_M is worth testing; we saw about 3% quality improvement on our benchmark suite with roughly 12% more memory.

Machine learning model selection matters more than hardware tuning. The biggest performance gains we found weren't from tuning llama.cpp parameters — they came from switching models. Qwen2-57B-A14B outperforms Mixtral 8x7B on generation speed while using comparable memory. If you're benchmarking local AI inference for MoE models, test multiple architectures before optimizing your inference stack.

Three Things to Take Away

If you're evaluating local MoE inference for production use, here's what our data says:

One: Unified memory architectures like Strix Halo are the right hardware bet for MoE models. The elimination of PCIe bandwidth as a bottleneck matters more than raw GPU compute for generation-phase throughput.

Two: ROCm's APU support is not production-ready as of mid-2026. Vulkan is stable, ships with AMD's maintained drivers, and performs within 15–20% of a theoretical well-tuned ROCm path — that's an acceptable trade for reliability. Build with -DGGML_VULKAN=ON and move on.

Three: Qwen2-57B-A14B is currently the best large MoE model for local AI inference on this class of hardware. Better generation speed than Mixtral, comparable memory footprint, stronger reasoning on structured tasks. DeepSeek-V2-Lite is the right choice when you need real-time latency and can accept a quality ceiling.

We'll publish the full benchmark scripts and model configs on the Bedda.tech GitHub as this work matures. If you're running similar hardware or trying to replicate these numbers, reach out — the Strix Halo community is small enough that everyone doing serious work on it should be sharing data.