Mega4alik/ollm

Runs 405B LLMs on 8GB VRAM

AirLLM 2.8T LLM inference with single 4GB GPU

Smart inference engine that adapts to your hardware to run models faster, even when they don’t fit in memory

maxxing the ability to run different LLMs on the smallest devices possible

~1300 line, minimal, extensible LLM inference engine built from scratch.

A fast inference library for running LLMs locally on modern consumer-class GPUs

KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM

KVMem virtualizes KV memory across inference backends to make ultra-long-context local AI accessible on limited hardware.

A high-throughput and memory-efficient inference and serving engine for LLMs