sahibzada-allahyar/Obsidian-Memory-Transformer

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

RWKV-X is a Linear Complexity Hybrid Language Model based on the RWKV architecture, integrating Sparse Attention to improve the model's long sequence processing capabilities.

Long Context Pre-Training with Lighthouse Attention

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining

[ICML'24 Spotlight] LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

MoBA: Mixture of Block Attention for Long-Context LLMs

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

Trainable fast and memory-efficient sparse attention

Breaking the quadratic memory barrier in AI