MoonshotAI/Moonlight

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Go ahead and axolotl questions

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

Minimalistic large language model 3D-parallelism training

A simple, performant, and scalable Jax LLM!

Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

NanoGPT (124M) in 90 seconds

AllenAI's post-training codebase