thc1006/qwen3.6-speculative-decoding-rtx3090

One llama.cpp flag unlocks +33-39% decode speed for Qwen3.8-27B on consumer GPUs. The MTP head already ships inside your GGUF. Recipe, paired benchmarks, probe tool.

Qwen3.8-27B EXL3 (4.00 bpw) + DFlash2 speculative decoding for ExLlamaV3, validated at 262k context on a 24 GB RTX 3090

Speculative decoding has plenty of tricks. Which ones actually make LLMs faster? Explore 7 methods, 26 checkpoints, and a shared training and evaluation framework.

A llama.cpp fork with TurboQuant/TCQ KV cache, fused Qwen MTP speculative decoding, and multi-GPU long-context serving fixes.

Deployment kit: Qwen3.8-27B EXL3 3.5bpw target + DFlash2 EXL3 5.0bpw speculative draft — launcher, config, model cards

A lightweight inference engine supporting speculative speculative decoding (SSD).

NInfer port for Qwen 3.8 27B on RTX 4090