Evans
Running models that don't fit. Writing down what it costs.
I’m Evans. I run large mixture-of-experts models on a dual-Xeon box with two RTX 3090s and 192 GB of DDR4. The expert weights live in host memory, everything else on the GPUs, and the interesting work is in the gap between the two.
What I publish is measurements: what a change did to decode and prefill, what broke, and what it took to find out. The code I’ve sent upstream to llama.cpp came out of the same runs.
Decode on Qwen3.8-Flash-Next is where I wanted it. Prefill with the experts in host RAM is the open problem.
Keeps an explicit -ot host buffer override from being silently downgraded to a plain CPU buffer under mmap, and reads pinned tensors from the file instead of faulting them through the mapping. Prefill 166 to 379 t/s with 40 expert layers on the host; load 512 s to 168 s.
17 to 41 t/s decode through the expert cache, UD-Q4_K_XL, and MTP speculative decoding. Next: a proper comparison against Qwen3.8-27B at Q8, speed and quality, on the same box.
- Note
A silent -ot downgrade under mmap, and a 9-line loader fix
Why -ot ...=CUDA_Host did nothing on master, what my PR #28223 changes, and where the 8 minutes of load time went.
- Part 02
Qwen3.8-Flash-Next on 2x3090 + DDR4, part 2: 25-29 to 37-41 t/s with UD-Q4_K_XL, the expert cache, and MTP
A quant swap, MTP on top of the cache, a 4x faster load, a bug in the cache PR, and the discovery that my RAM had been thermal throttling the whole time.
- Part 01
Qwen3.8-Flash-Next on 2x3090 + DDR4: 17 to 25-29 t/s decode with the expert cache PR
All 48 expert layers in host RAM, an LRU cache of hot experts in VRAM, and the VRAM budget that made it pay.
Can query-selected KV paging fit a 262k context on a single 24GB GPU? Measured on Qwen3.6-27B: no. Even a perfect oracle selector falls far short of the attention mass needed. Harness plus the negative result, so nobody else has to run it.
Downloads that finish. A supervisor around aria2c for huge model files. Survives stalls, expired links, and reboots without losing progress. SHA256-verified. Moved about 200GB with it.
Self-hosted personal organiser you talk to. Whisper, a local LLM, SQLite, FastAPI, and a React PWA. Nothing leaves your machine. Runs end to end on 12GB of VRAM.
