Brainchip-Inc/Do-Transformers-Need-3-Projections

[NeurIPS 2025 Spotlight] TPA: Tensor ProducT ATTenTion Transformer (https://arxiv.org/abs/2501.06425)

A place to store reusable transformer components of my own creation or found on the interwebs

RoPE attention is an exact forward-pass gradient step with softmax intact

⚡ Official code for “Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus”.

Explorations into the recently proposed Taylor Series Linear Attention

Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"

Official Project Page for Deep Delta Learning (https://arxiv.org/abs/2601.00417)