lucienhuangfu/eLLM

A sparse-first inference engine (SparseEngine).

RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

Sparse Decoupled Attention for Efficient Long-Context LLM Inference

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

[VLDB 26, NeurIPS 25] Scalable long-context LLM decoding that leverages sparsity—by treating the KV cache as a vector storage system.

KVMem virtualizes KV memory across inference backends to make ultra-long-context local AI accessible on limited hardware.

Efficient AI Inference & Serving