JIA-Lab-research/LongLoRA

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

Code and documentation to train Stanford's Alpaca models, and generate the data.

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

MoBA: Mixture of Block Attention for Long-Context LLMs

AllenAI's post-training codebase

中文LLaMA-2 & Alpaca-2大模型二期项目 + 64K超长上下文模型 (Chinese LLaMA-2 & Alpaca-2 LLMs with 64K long context models)

Large World Model -- Modeling Text and Video with Millions Context