jesdga95/basalt

High-speed Large Language Model Serving for Local Deployment

Run Qwen3.8-Flash-Next (180B) on a single DGX Spark by keeping its 51B n-gram embedding table on NVMe

One line of NInfer for the RTX 3090, RTX 4090 and RTX 5090, consolidated from the forks that carry it: GGUF block formats, ternary Bonsai 2, device route profiles, MTP and DFlash2.

High-performance single-GPU inference for selected model checkpoints and GPUs.

High-performance single-GPU inference for selected model checkpoints and GPUs.

Hyper optimised Qwen3.8-27B inference on one RTX 3090: ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds.

Strata minus the bullshit. clone, rsync to prod, compile and go

Qwen3.8-27B on 16-32 GB Nvidia GPUs one-click install for Windows / Linux

From-scratch C++/CUDA inference engine for Qwen3-8B