MoonshotAI/Attention-Residuals

[Technical Report] Multi-Stream Attention Residuals

This repo is a alternative solution for standard self-attention. It uses tree routing with learnable functions for cheaper self-attention while not compromising accuracy.

[CVPR 2026 Highlight] Official implementation of Log-linear Sparse Attention (LLSA).

SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse–Linear Attention

Explorations into the recently proposed Taylor Series Linear Attention

Official Project Page for Deep Delta Learning (https://arxiv.org/abs/2601.00417)

SSOG-Attention: Near-linear Visual-Attention and competitive results on ImageNet.

RoPE attention is an exact forward-pass gradient step with softmax intact

Implementation of fused cosine similarity attention in the same style as Flash Attention