mit-han-lab/streaming-llm

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

Memory Sparse Attention - A scalable, end-to-end trainable latent-memory framework for 100M-token contexts.

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Unified KV Cache Compression Methods for Auto-Regressive Models

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

AirLLM 70B inference with single 4GB GPU

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)