natpate/ninfer-windows

High-performance single-GPU inference for selected model checkpoints and GPUs.

High-performance single-GPU inference for selected model checkpoints and GPUs.

Hyper optimised Qwen3.8-27B inference on one RTX 3090: ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds.

NInfer port for Qwen 3.8 27B on RTX 4090

Qwen3.8-27B on one RTX 4090: full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention prefill, vision, llama.cpp-compatible /metrics + /slots

High-speed Large Language Model Serving for Local Deployment

~1300 line, minimal, extensible LLM inference engine built from scratch.

Fastest LLM inference runtime for Apple Silicon

High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI