qlabs-eng/dust

Speedrun for pretraining without backpropagation

Educational Transformer implementation from scratch - no PyTorch, no TensorFlow, just NumPy

Code implementation from my blog post: https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch

DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation

A concise but complete full-attention transformer with a set of promising experimental features from various papers

Official codebase for "Next-Latent Prediction Transformers Learn Compact World Models"

Well documented, unit tested, type checked and formatted implementation of a vanilla transformer - for educational purposes.

Multi-framework implementation of Deep Kernel Shaping and Tailored Activation Transformations, which are methods that modify neural network models (and their initializations) to make them easier to train.