Helldez/BigMoeOnEdge

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

Run MoE models bigger than your RAM, without the speed cliff

Running a big model on a small laptop

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

LLM inference in C/C++

A full, unpruned 177B MoE model at 11.5 tok/s on one 12 GB GPU. Expert streaming for llama.cpp, with the measurement harness and every dead end.

Run large Mixture-of-Experts language models well on the memory you have.

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory