facebookresearch/schedule_free

optimizer & lr scheduler & loss function collections in PyTorch

Efficient optimizers

For optimization algorithm research and development.

An efficient implementation of learned optimizers in PyTorch

Dion optimizer algorithm

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

Gefen: Optimized Stochastic Optimizer

A PyTorch Lightning extension that accelerates and enhances foundation model experimentation with flexible fine-tuning schedules.