musicurgy/delta-inference-engine

Native DeepSeek V4.1 Flash inference across one RTX PRO 6000 and four DGX Spark expert workers

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) on 2x RTX 3090 + 128 GB RAM. Rust + CUDA, T16 quantization. 40+ tok/s measured with expert deferral + FP8 projections.

Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark, and full continuous batch support

High-performance single-GPU inference for selected model checkpoints and GPUs.

Windows-first OpenAI-compatible local LLM server powered by OpenVINO GenAI for Intel CPU/GPU/NPU, with chat UI, model conversion, and setup scripts.

CUDA Blackwell performance fork of antirez/ds4 (DeepSeek-V4-Flash): ~4x prefill, ~1.5x decode, continuous batching, weight server, DSpark

DeepSeek 4 Flash local inference engine for Metal and CUDA with M5 optimizations.

Ollama-compatible LLM server on a tuned llama.cpp fork. Beats vllm by 40% on single stream decode, and by 20% on multi stream decode, and beats ollama by much 2-4x. Made possible with custom CUDA kernels, MTP, speculation and its own 3-bit builds. Uncensors models in one command. Windows and Linux, NVIDIA. Optimized to not eat up ram.