datamllab/LongLM

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining

Recursive Language Models for efficient long-context processing. Analyze 1M+ tokens by storing context in a Python REPL while reducing LLM token usage.

LLM KV cache compression made easy

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Optimizing inference proxy for LLMs

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.