Local LLM engine for Windows and one NVIDIA RTX 5090, built for Qwen3.8-27B
Haru: a low-latency, multilingual, full-duplex spoken dialogue model
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
The open engine for running AI on your own computer. Fast, exact, open source.
One-click & interactive deployment of Qwen3.8-Flash-Next-CYBERSECURITY-NVFP4 (262K context + MTP NEXTN) on Google Colab G4 GPU (RTX PRO 6000 Blackwell 96GB)
Disaggregated Prefill-Decode LLM serving engine on mini-sglang with a proved KV page state machine (PdLedger)
A inference framework that adapt to diverse and dynamic agent workload
Volta Max uPLIft: DeepSeek-V4.1-Flash-1017 on 8x V100-SXM2 (2018) with as much of the routed work at NVIDIA's NVFP4 precision as fits and every routed expert on the cards. One Hugging Face package, the code, every measurement. PLI Labs.
Private chat and code on your Mac. Run a 177B model on a 128 GB M5 Max: local Qwen3.8 Flash-Next inference with a 65,536-token default or optional 131,072-token context, Tokspur's own Metal engine with native MTP or DFlash2 drafting, and an OpenAI-compatible API. Apache 2.0.
Compress any LLM in seconds, load it for inference and in one click you have an AI that could have taken hours of vLLM configuration and what not to setup and load. Of course there are opensource compressed models but this is for people who have build their own AI, finetuned it, etc.
GPT-2 small (124M) in hand-written AArch64 assembly. No libc calls. About 100 tokens/s on an M1.
A Rust inference engine currently targeting AMD R9700 GPU's. buymeacoffee.com/bkuyper
DeepSeek V4.1 Flash on 4x DGX Spark: 438 tok/s at 16 users, ~80/138 single-user, 1M context. Fork of blaesus/dsgx10.
GLM-5.3-Flash on DGX Sparks (ver 1104) with TensorFold
maxxing the ability to run different LLMs on the smallest devices possible
Native Apple Silicon inference engine for many AI agents at once: C++/Metal, agent-contexts API, 8-bit paged KV, SSD tier (Qwen3.8)
DeepSeek-V4.1-Flash-1017 on 8x V100-SXM2 (2018): 119.0 tok/s decode, 120.8 thinking on (measured), 524K context, 300K-token first token in 178 s. PP2xTP4 port of 1Cat-vLLM #1277 with every patch, measurement and lost lever. PLI Labs.
SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding