mustafaaljadery/gemma-2B-10M

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining

Recursive Language Models for efficient long-context processing. Analyze 1M+ tokens by storing context in a Python REPL while reducing LLM token usage.

MoBA: Mixture of Block Attention for Long-Context LLMs

[ICML'24 Spotlight] LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

Memory Sparse Attention - A scalable, end-to-end trainable latent-memory framework for 100M-token contexts.

Official repo for "Make Your LLM Fully Utilize the Context"