Of the novelty-gated ideas, the best one improved on the 3500 baseline by only 125 steps compared to PR-copying which gives 500+ step gains.

fun little artifact, i worked on something similar to freon last year and started writing an (unedited) post that is hidden on my blog: very naive implementation of steepest descent under various p…

Modded-NanoGPT optimization result #14 (2026/05/04): @Sam_Acqua has achieved a new record of 3150 steps (-60), by adding SOAP preconditioning before Muon orthogonalization for the MLP weights…