marzukia/qMLX

Extreme weight + KV cache compression for LLMs on Apple Silicon (MLX implementation of Google's TurboQuant)

Qwen3.8-Flash-Next on Apple Silicon: an MLX engine that reads Strata's GGUF files and speaks its line protocol (unofficial)

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Fork of antirez/ds4 for Apple Silicon, tuned mostly for M1/M2: Qwen3.8-Flash-Next with 262K context on 64 GB and up, 35 tok/s on an M1 Max, MTP and vision, one-line install.

Swift + Metal MoE inference for Apple Silicon: Qwen 3.6 35B at 23.5–29.3 tok/s decode with 2.20× faster long-prompt prefill on a 24 GB M5; Gemma 4 26B in ~2 GB, DeepSeek-V4-Flash 284B, Inkling-Small 276B, native Mac app, CLI, and OpenAI-compatible server.

⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.

W8A8/W4A8 inference + optimized SDPA on Apple Silicon — unlocking unused INT8 TensorOps in M5 for 1.2–1.9× faster LLM prefill, plus FlashInfer-inspired GQA decode attention for up to 1.6× SDPA speedup, built as MLX custom primitives.

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

Low-level Metal inference engine for Apple M4: universal quant formats at MLX parity, 1M-token out-of-core contexts, and speculative decode off flash storage.