davide-troiani/moeshift

Run large Mixture-of-Experts language models well on the memory you have.

Running a big model on a small laptop

LLM inference in C/C++

Swift + Metal MoE inference for Apple Silicon: Qwen 3.6 35B at 23.5–29.3 tok/s decode with 2.20× faster long-prompt prefill on a 24 GB M5; Gemma 4 26B in ~2 GB, DeepSeek-V4-Flash 284B, Inkling-Small 276B, native Mac app, CLI, and OpenAI-compatible server.

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

Inferno aims to be a super lightweight, highly efficient Rust inference engine for running open weights models on Apple Silicon with Metal, targeting machines such as a MacBook Pro with 64 GB of unified memory.

Flash-MoE sidecar slot-bank runtime for large GGUF MoE models on Apple Silicon — llama.cpp fork

works well with rocks; native Apple Silicon inference engine for capable and tiny models (ternary, Qwen) via MLX Swift. Hadamard-rotated KV cache, vision, tool calling, OpenAI + Anthropic APIs