wici-ai/inferssd

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

Run the full 2.78-trillion-parameter Kimi K3 model, DeepSeek V4.1 Flash or GLM-5.3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

Run 70B+ LLMs on Apple Silicon by using SSD as extended memory — intelligent layer streaming and caching for Mac

Run multiple resource-heavy Large Models (LM) on the same machine with limited amount of VRAM/other resources by exposing them on different ports and loading/unloading them on demand

High-speed Large Language Model Serving for Local Deployment

The official API server for Exllama. OAI compatible, lightweight, and fast.

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

Run GreenBitAI's Quantized LLMs on Apple Devices with MLX

P2P distributed LLM inference on browsers