Epistates/pmetal

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Fine-tune and run LLMs locally on your M-series Mac. A powerful desktop interface built on Apple's MLX framework for zero-setup AI.

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

Fast, exact LLM decoding on Apple Silicon (MLX) behind an OpenAI-compatible endpoint

W8A8/W4A8 inference + optimized SDPA on Apple Silicon — unlocking unused INT8 TensorOps in M5 for 1.2–1.9× faster LLM prefill, plus FlashInfer-inspired GQA decode attention for up to 1.6× SDPA speedup, built as MLX custom primitives.

A native Mac App for LLM fine-tuning on Apple Silicon — fully on-device, fully open source.

Running a big model on a small laptop

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.