arizqi/cpubrrr

Inferno aims to be a super lightweight, highly efficient Rust inference engine for running open weights models on Apple Silicon with Metal, targeting machines such as a MacBook Pro with 64 GB of unified memory.

Fastest LLM inference runtime for Apple Silicon

Our lightning-fast inference engine.

lithos-metal: high-performance LLM inference kernels and serving for Apple silicon.

Running a big model on a small laptop

Swift + Metal MoE inference for Apple Silicon: Qwen 3.6 35B at 23.5–29.3 tok/s decode with 2.20× faster long-prompt prefill on a 24 GB M5; Gemma 4 26B in ~2 GB, DeepSeek-V4-Flash 284B, Inkling-Small 276B, native Mac app, CLI, and OpenAI-compatible server.

A rust mlx server, ui, and model router for fast, dependency free inference on apple architecture

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Pure C++23 LLM inference for Apple Silicon chips