howard-hou/RWKV-X

Light-weight infrastructure for express LLM pre-training

Open-source toolkit for training, Priming, and serving next generation Hybrid architectures

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

A novel LLM architecture written in highly optimized low-level C++/CUDA with a new Long-Term Memory (LTM) mechanism for large context windows.

Trainable fast and memory-efficient sparse attention

MoBA: Mixture of Block Attention for Long-Context LLMs

Memory Sparse Attention - A scalable, end-to-end trainable latent-memory framework for 100M-token contexts.

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention