QuixiAI/grokadamw

Gefen: Optimized Stochastic Optimizer

Efficient optimizers

For optimization algorithm research and development.

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

Implementation of the proposed Adam-atan2 from Google Deepmind in Pytorch

Dion optimizer algorithm

optimizer & lr scheduler & loss function collections in PyTorch