AbdelrhmanEbied/Tiny-MoE

Tutel MoE: Optimized Mixture-of-Experts Library, Support GLM/Kimi/Gpt/DeepSeek/Qwen using FP8/NVFP4/MXFP4

Implementation of LatentMoE,Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (Elango et al., NVIDIA 2026) — in Pytorch. A single-file, dependency-light layer you can drop in place of a standard MoE FFN.

Instella MoE is a fully open state-of-the-art Mixture-of-Experts language model, trained end-to-end on AMD Instinct™ GPUs.

Readable, composable PyTorch library for language models: build Llama, Qwen, Gemma, DeepSeek, Kimi and 20+ other architectures from plain nn.Modules, then train them on CPU, one GPU, or multi-GPU.

A 124.6M LLM trained from scratch on 13B tokens on a single RTX 4090 — tokenizer, pretraining, post-training, evaluation, and reproducible inference.

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

A 49M-parameter Kimi K3-inspired language model trainable on one 8GB GPU

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model