midagedev/bloomery

LLM inference in C/C++

A custom fork for frontier-scale mixture-of-experts inference on an RTX 3060 with 12 GB VRAM, 16 GB system RAM and local NVMe.** Betelgeuse keeps dense computation on the GPU and brings routed experts into a bounded VRAM pool only when needed.

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

High-speed Large Language Model Serving for Local Deployment

Memory-aware GGUF inference runtime with OpenAI-compatible serving

A high-throughput and memory-efficient inference and serving engine for LLMs - Optimized for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60