Writeups
Benchmarks, failure notes, and the commands to reproduce them.
- Note
A silent -ot downgrade under mmap, and a 9-line loader fix
Why -ot ...=CUDA_Host did nothing on master, what my PR #28223 changes, and where the 8 minutes of load time went.
- Part 02
Qwen3.8-Flash-Next on 2x3090 + DDR4, part 2: 25-29 to 37-41 t/s with UD-Q4_K_XL, the expert cache, and MTP
A quant swap, MTP on top of the cache, a 4x faster load, a bug in the cache PR, and the discovery that my RAM had been thermal throttling the whole time.
- Part 01
Qwen3.8-Flash-Next on 2x3090 + DDR4: 17 to 25-29 t/s decode with the expert cache PR
All 48 expert layers in host RAM, an LRU cache of hot experts in VRAM, and the VRAM budget that made it pay.