Running 96GB Local AI on a Tablet: The Hardware Hacks That Actually Work
Running 96GB Local AI on a Tablet: The Hardware Hacks That Actually Work
TL;DR: One of our nodes is an ASUS ROG Flow Z13 — yes, the detachable tablet — running Fedora Linux on an AMD Ryzen AI MAX+ 395 ("Strix Halo") with 128GB of unified memory. We fixed a BIOS default that reserved almost all of it for graphics, worked around a broken ROCm build on this GPU by switching Ollama to Vulkan, and got bitten by a newer Ollama silently dropping the GPU and falling back to CPU. End state: a 90GB local model loads in 31 seconds and serves 27-39 tokens/second, and our best MoE model hits 67.9 tok/s. No cloud API involved.
Why a Tablet?
Most of our inference runs on cloud APIs and a couple of dedicated GPU boxes. But we also run one node, codename strix-halo, on hardware most people assume can't do real local AI: a 13.3" 2-in-1 tablet.
The ASUS ROG Flow Z13 (2025) ships with:
- AMD Ryzen AI MAX+ 395 ("Strix Halo") — Zen 5 CPU + RDNA 3.5 iGPU on one die
- 128GB unified LPDDR5X, shared between CPU and GPU
- AMD Radeon 8060S iGPU (gfx1151)
- USB-C dock for desktop-mode use
We wiped Windows and put Fedora 43 Workstation on it. What follows is what it took to get that unified memory actually usable for inference, not what the spec sheet implies.
Step 1: The BIOS Trap
Unified memory means the GPU doesn't have its own VRAM — it borrows a slice of system RAM, and firmware decides how much. On this board, the default UMA Frame Buffer setting is 512MB. That's a display-only number; it's nowhere close to enough to load anything.
The fix is in BIOS, not software: raise the UMA Frame Buffer allocation to its max, 96GB. On a fresh install this is the single setting that turns the Z13 from "laptop with a decent iGPU" into "128GB inference box." Miss it and every model load just OOMs with no useful error.
With that set and ROCm installed, rocminfo reported 96.5GB VRAM available to the GPU — the number in this post's title, and the one we shipped with in February.
Step 2: ROCm Looked Right, Then Broke — Vulkan Won
AMD's own stack should be the obvious choice for an AMD GPU. We got ROCm 7.1.1 installed from Fedora's repos (Fedora ships it under /usr, not the /opt/rocm path most guides assume — first correction to make to any ROCm tutorial on this machine).
It didn't hold up. gfx1151 (this exact iGPU) hits a broken hipStreamCreateWithFlags path in ROCm, and rocwmma doesn't support gfx1151 at all on the ROCm version we had — we had to strip that build flag entirely.
The fix that actually worked: run Ollama on its Vulkan backend instead of ROCm.
Environment=OLLAMA_VULKAN=1
Environment=OLLAMA_LLM_LIBRARY=vulkan
Vulkan compute doesn't touch ROCm's broken paths at all, and on this GPU it's the backend that's actually stable in production, not just the fallback.
Step 3: The Gotcha That Cost Us Silent CPU Inference
Seven months later, in September, we came back to upgrade Ollama (0.17.2 → 0.32.14) and hit a regression that's worth calling out because it's easy to miss: Ollama 0.32+ drops integrated GPUs by default. No error — it just silently falls back to CPU. The only trace is one line in the journal:
dropping integrated GPU; to enable, set OLLAMA_IGPU_ENABLE=1
Fix is one env var in the systemd drop-in:
Environment=OLLAMA_IGPU_ENABLE=1
After that, the device shows up correctly:
Vulkan0 AMD Radeon 8060S (RADV GFX1151) type=iGPU total=128.0 GiB available=126.3 GiB
That 126.3GB available (vs. the 96.5GB we started with in February) is the same physical machine — later tuning and a newer driver stack recovered more of the unified pool. If you're auditing a unified-memory AMD box after any Ollama upgrade, checking that "inference compute" journal line is now a standing step for us, not a one-time fix.
Step 4: MoE Beats Dense on This Hardware — By A Lot
Strix Halo's unified memory is generous but its bandwidth is the real constraint, so the model architecture matters more here than on a discrete GPU.
Dense model, bandwidth-bound:
hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M → 12.8 tok/s gen
A 17GB dense model at roughly 256GB/s effective bandwidth — this is expected and matches the roofline, not a bug.
MoE model, same hardware:
qwen3.6:35b-a3b (MoE) → 67.9 tok/s gen, zero crashes
That MoE number used to fail outright — this exact model class SIGSEGV'd on every request back when we were still on the older Ollama/Vulkan combo (a known MoE-on-Vulkan crash). The September Ollama upgrade fixed it; the hardware was never the problem. That's the actual lesson: when a workaround claims a "hardware limitation," check the software version first. Our earlier February numbers told the same story before the crash regression ever showed up — qwen3:30b (MoE, ~3B active) ran ~72 tok/s vs. ~5 tok/s for a dense 70B on the same box.
Step 5: 90GB in 31 Seconds — The Big One
The real stress test was a 90GB GGUF from the DeepSeek-V4 "Flash Next" family, quantized UD-Q3_K_XL, split across 3 shards. We built a Vulkan fork of llama.cpp for this (halo-box/strix-llama.cpp) since we needed --ngram-on-disk caching and current Vulkan compute kernels together.
Service config, for anyone reproducing this:
--ngram-on-disk --ngram-cache 2048 -c 131072 --parallel 2 \
-ngl all -fa on --jinja -ctk q8_0 -ctv q8_0 \
--alias qwen3.8-flash-next --metrics
Results, measured, not estimated:
Load time: 31s
Resident (cgroup): 41GB
Single-stream: 27 tok/s
2 parallel streams: 39 tok/s aggregate
Prefill @ 4k context: 354 tok/s
Tool calls: correct, --jinja template
Worth noting: this box isn't a bare test rig. It's also running a self-hosted Sentry stack (~10GB across two JVMs + ClickHouse), stirling-pdf, and a normal desktop session — call it 40GB of baseline usage before any model loads. The 90GB model still fit comfortably inside the 126GB pool with that overhead included.
What We Learned
1. The bottleneck is memory bandwidth, not memory size. 128GB of unified memory sounds unlimited, but the dense-vs-MoE gap (12.8 vs 67.9 tok/s on comparably-sized models) is a bandwidth story, not a capacity one. If you're picking a model for unified-memory hardware, pick architecture first.
2. "Broken on this hardware" is often "broken on this software version." The MoE crash that looked like a Vulkan/gfx1151 limitation for months was an Ollama bug, gone after one version bump. Re-test your "known limitations" list before you write them into a blog post.
3. BIOS settings are load-bearing for unified memory. The single highest-leverage fix on this whole machine — UMA Frame Buffer 512MB → 96GB — happens before Linux even boots. No amount of driver tuning fixes a firmware default.
4. Check the actual GPU-detection log line after every inference-stack upgrade. Silent CPU fallback doesn't throw an error; it just gets slower and you might not notice for a while if you're not watching throughput.
5. Fedora + AMD unified memory needs path corrections, not just install steps. /usr instead of /opt/rocm, rocwmma unsupported on gfx1151, HIP_VISIBLE_DEVICES overrides that break auto-detection instead of helping it — every generic ROCm guide needed at least one correction on this exact chip.
The Real Use Case
We didn't do this to win a benchmark. strix-halo is one node in our agent fleet's local-inference pool, load-balanced against another local box via LiteLLM's least-busy routing. It handles coder and general-purpose model traffic alongside our other GPU node, and it's proof that a genuinely portable device — the kind you'd actually carry, not a rack — can carry real inference weight without a cloud dependency.
For a company building distributed AI agents, that matters: it's one more class of hardware our fleet can run on, and one more thing we don't have to send to someone else's API.
Oliver's Lab. Real specs, real bugs, real numbers — from hardware we actually run in production.
Interested in edge AI or local inference? We're hiring. Apply to BeddaTech.