krafton-ai/moe-to-dense

The official implementation of the paper "Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques (TMLR)".

Expert Specialized Fine-Tuning

An Open Source Toolkit For LLM Distillation

Instella MoE is a fully open state-of-the-art Mixture-of-Experts language model, trained end-to-end on AMD Instinct™ GPUs.

Codebase for Merging Language Models (ICML 2024)

Tutel MoE: Optimized Mixture-of-Experts Library, Support GLM/Kimi/Gpt/DeepSeek/Qwen using FP8/NVFP4/MXFP4

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation (DN-MOPD)

Tiny-MoE is a lightweight Mixture-of-Experts language model built entirely from scratch in native PyTorch and trained end-to-end on Kaggle using free 2× NVIDIA T4 GPUs. The project implements modern LLM techniques—including MLA, RoPE, YaRN, streaming pre-training, and efficient inference—without relying on existing model implementations.