ighoshsubho/lighthouse-attention

MoBA: Mixture of Block Attention for Long-Context LLMs

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

Official code for HiLS-Attention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

A novel LLM architecture written in highly optimized low-level C++/CUDA with a new Long-Term Memory (LTM) mechanism for large context windows.

Advancing the frontier of efficient AI

A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Training

Trainable fast and memory-efficient sparse attention

RWKV-X is a Linear Complexity Hybrid Language Model based on the RWKV architecture, integrating Sparse Attention to improve the model's long sequence processing capabilities.