Belilovsky-Lab/pylo

optimizer & lr scheduler & loss function collections in PyTorch

Efficient optimizers

Gefen: Optimized Stochastic Optimizer

For optimization algorithm research and development.

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

Schedule-Free Optimization in PyTorch

Dion optimizer algorithm

An industrial extension library of pytorch to accelerate large scale model training