Yifei-Zuo/Parallax

We will release the training kernels soon 😁

Parallax achieves higher compute density than softmax attention during the decoding stage, where we co-design the algorithm and system to attain both higher model performance and efficiency. Check…

For me, the coolest finding is that Muon optimizer is crucial for Parallax to move beyond Softmax Attention. Lesson — don't evaluate new architectures solely under AdamW, you'll miss the good ones.…

~6/7~ Crucially, we find Muon counterfactually amplifies the advantage of Parallax. The strength of Parallax depends heavily on the norm and alignment of the probe and the KV covariance, which is…