jitllm/jitllm

High-speed Large Language Model Serving for Local Deployment

Get more intelligence from every bit. Better quantization formats and smarter calibration let larger, stronger models run smoothly on the hardware you already own.

Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.

Zig INferenCe Engine — Local LLM inference on AMD GPUs and Apple Silicon

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

Atlas Inference is a dependency free lightweight inference engine meant to run LLMs on YOUR compute seamlessly. Supports GB10 and Strix Halo at the moment, hoping to expand to more!

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

A high-throughput and memory-efficient inference and serving engine for LLMs