rayl15/Godwit

Swift + Metal MoE inference for Apple Silicon: Qwen 3.6 35B at 23.5–29.3 tok/s decode with 2.20× faster long-prompt prefill on a 24 GB M5; Gemma 4 26B in ~2 GB, DeepSeek-V4-Flash 284B, Inkling-Small 276B, native Mac app, CLI, and OpenAI-compatible server.

Running a big model on a small laptop

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

DeepSeek-V4-Flash-0731 284B inference in ~25 GB of RAM / Qwen3.8-Next-Flash inference in ~15 GB of RAM on any M-series MacBook

Inferno aims to be a super lightweight, highly efficient Rust inference engine for running open weights models on Apple Silicon with Metal, targeting machines such as a MacBook Pro with 64 GB of unified memory.

Big models. Small Macs. Zero excuses.

macMLX brings local LLM inference to Apple Silicon with a first-class native macOS experience. No cloud, no telemetry, no Electron — just your Mac running models at full speed.

LLM inference in C/C++

Run models too big for your Mac's memory