base on A general fine-tuning kit geared toward diffusion models. # SimpleTuner 💹 > ℹ️ No data is sent to any third parties except through opt-in flag `report_to`, `push_to_hub`, or webhooks which must be manually configured. **SimpleTuner** is geared towards simplicity, with a focus on making the code easily understood. This codebase serves as a shared academic exercise, and contributions are welcome. If you'd like to join our community, we can be found [on Discord](https://discord.gg/JGkSwEbjRb) via Terminus Research Group. If you have any questions, please feel free to reach out to us there. ## Table of Contents - [Design Philosophy](#design-philosophy) - [Tutorial](#tutorial) - [Features](#features) - [Core Training Features](#core-training-features) - [Model Architecture Support](#model-architecture-support) - [Advanced Training Techniques](#advanced-training-techniques) - [Model-Specific Features](#model-specific-features) - [Quickstart Guides](#quickstart-guides) - [Hardware Requirements](#hardware-requirements) - [Toolkit](#toolkit) - [Setup](#setup) - [Troubleshooting](#troubleshooting) ## Design Philosophy - **Simplicity**: Aiming to have good default settings for most use cases, so less tinkering is required. - **Versatility**: Designed to handle a wide range of image quantities - from small datasets to extensive collections. - **Cutting-Edge Features**: Only incorporates features that have proven efficacy, avoiding the addition of untested options. ## Tutorial Please fully explore this README before embarking on [the tutorial](/documentation/TUTORIAL.md), as it contains vital information that you might need to know first. For a quick start without reading the full documentation, you can use the [Quick Start](/documentation/QUICKSTART.md) guide. For memory-constrained systems, see the [DeepSpeed document](/documentation/DEEPSPEED.md) which explains how to use 🤗Accelerate to configure Microsoft's DeepSpeed for optimiser state offload. For multi-node distributed training, [this guide](/documentation/DISTRIBUTED.md) will help tweak the configurations from the INSTALL and Quickstart guides to be suitable for multi-node training, and optimising for image datasets numbering in the billions of samples. --- ## Features SimpleTuner provides comprehensive training support across multiple diffusion model architectures with consistent feature availability: ### Core Training Features - **Multi-GPU training** - Distributed training across multiple GPUs with automatic optimization - **Advanced caching** - Image, video, and caption embeddings cached to disk for faster training - **Aspect bucketing** - Support for varied image/video sizes and aspect ratios - **Memory optimization** - Most models trainable on 24G GPU, many on 16G with optimizations - **DeepSpeed integration** - Train large models on smaller GPUs with gradient checkpointing and optimizer state offload - **S3 training** - Train directly from cloud storage (Cloudflare R2, Wasabi S3) - **EMA support** - Exponential moving average weights for improved stability and quality ### Model Architecture Support | Model | Parameters | PEFT LoRA | Lycoris | Full-Rank | ControlNet | Quantization | Flow Matching | Text Encoders | |-------|------------|-----------|---------|-----------|------------|--------------|---------------|---------------| | **Stable Diffusion XL** | 3.5B | ✓ | ✓ | ✓ | ✓ | int8/nf4 | ✗ | CLIP-L/G | | **Stable Diffusion 3** | 2B-8B | ✓ | ✓ | ✓* | ✓ | int8/fp8/nf4 | ✓ | CLIP-L/G + T5-XXL | | **Flux.1** | 12B | ✓ | ✓ | ✓* | ✓ | int8/fp8/nf4 | ✓ | CLIP-L + T5-XXL | | **Auraflow** | 6.8B | ✓ | ✓ | ✓* | ✓ | int8/fp8/nf4 | ✓ | UMT5-XXL | | **PixArt Sigma** | 0.6B-0.9B | ✗ | ✓ | ✓ | ✓ | int8 | ✗ | T5-XXL | | **Sana** | 0.6B-4.8B | ✗ | ✓ | ✓ | ✗ | int8 | ✓ | Gemma2-2B | | **Lumina2** | 2B | ✓ | ✓ | ✓ | ✗ | int8 | ✓ | Gemma2 | | **Kwai Kolors** | 5B | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ChatGLM-6B | | **LTX Video** | 5B | ✓ | ✓ | ✓ | ✗ | int8/fp8 | ✓ | T5-XXL | | **Wan Video** | 1.3B-14B | ✓ | ✓ | ✓* | ✗ | int8 | ✓ | UMT5 | | **HiDream** | 17B (8.5B MoE) | ✓ | ✓ | ✓* | ✓ | int8/fp8/nf4 | ✓ | CLIP-L + T5-XXL + Llama | | **Cosmos2** | 2B-14B | ✗ | ✓ | ✓ | ✗ | int8 | ✓ | T5-XXL | | **OmniGen** | 3.8B | ✓ | ✓ | ✓ | ✗ | int8/fp8 | ✓ | T5-XXL | | **Qwen Image** | 20B | ✓ | ✓ | ✓* | ✗ | int8/nf4 (req.) | ✓ | T5-XXL | | **SD 1.x/2.x (Legacy)** | 0.9B | ✓ | ✓ | ✓ | ✓ | int8/nf4 | ✗ | CLIP-L | *✓ = Supported, ✗ = Not supported, * = Requires DeepSpeed for full-rank training* ### Advanced Training Techniques - **TREAD** - Token-wise dropout for transformer models, including Kontext training - **Masked loss training** - Superior convergence with segmentation/depth guidance - **Prior regularization** - Enhanced training stability for character consistency - **Gradient checkpointing** - Configurable intervals for memory/speed optimization - **Loss functions** - L2, Huber, Smooth L1 with scheduling support - **SNR weighting** - Min-SNR gamma weighting for improved training dynamics ### Model-Specific Features - **Flux Kontext** - Edit conditioning and image-to-image training for Flux models - **PixArt two-stage** - eDiff training pipeline support for PixArt Sigma - **Flow matching models** - Advanced scheduling with beta/uniform distributions - **HiDream MoE** - Mixture of Experts gate loss augmentation - **T5 masked training** - Enhanced fine details for Flux and compatible models - **QKV fusion** - Memory and speed optimizations (Flux, Lumina2) - **TREAD integration** - Selective token routing for Wan and Flux models - **Classifier-free guidance** - Optional CFG reintroduction for distilled models ### Quickstart Guides Detailed quickstart guides are available for all supported models: - **[Flux.1 Guide](/documentation/quickstart/FLUX.md)** - Includes Kontext editing support and QKV fusion - **[Stable Diffusion 3 Guide](/documentation/quickstart/SD3.md)** - Full and LoRA training with ControlNet - **[Stable Diffusion XL Guide](/documentation/quickstart/SDXL.md)** - Complete SDXL training pipeline - **[Auraflow Guide](/documentation/quickstart/AURAFLOW.md)** - Flow-matching model training - **[PixArt Sigma Guide](/documentation/quickstart/SIGMA.md)** - DiT model with two-stage support - **[Sana Guide](/documentation/quickstart/SANA.md)** - Lightweight flow-matching model - **[Lumina2 Guide](/documentation/quickstart/LUMINA2.md)** - 2B parameter flow-matching model - **[Kwai Kolors Guide](/documentation/quickstart/KOLORS.md)** - SDXL-based with ChatGLM encoder - **[LTX Video Guide](/documentation/quickstart/LTXVIDEO.md)** - Video diffusion training - **[Wan Video Guide](/documentation/quickstart/WAN.md)** - Video flow-matching with TREAD support - **[HiDream Guide](/documentation/quickstart/HIDREAM.md)** - MoE model with advanced features - **[Cosmos2 Guide](/documentation/quickstart/COSMOS2IMAGE.md)** - Multi-modal image generation - **[OmniGen Guide](/documentation/quickstart/OMNIGEN.md)** - Unified image generation model - **[Qwen Image Guide](/documentation/quickstart/QWEN_IMAGE.md)** - 20B parameter large-scale training --- ## Hardware Requirements ### General Requirements - **NVIDIA**: RTX 3080+ recommended (tested up to H200) - **AMD**: 7900 XTX 24GB and MI300X verified (higher memory usage vs NVIDIA) - **Apple**: M3 Max+ with 24GB+ unified memory for LoRA training ### Memory Guidelines by Model Size - **Large models (12B+)**: A100-80G for full-rank, 24G+ for LoRA/Lycoris - **Medium models (2B-8B)**: 16G+ for LoRA, 40G+ for full-rank training - **Small models (<2B)**: 12G+ sufficient for most training types **Note**: Quantization (int8/fp8/nf4) significantly reduces memory requirements. See individual [quickstart guides](#quickstart-guides) for model-specific requirements. ## Setup SimpleTuner can be installed via pip for most users: ```bash # Base installation (CPU-only PyTorch) pip install simpletuner # CUDA users (NVIDIA GPUs) pip install simpletuner[cuda] # ROCm users (AMD GPUs) pip install simpletuner[rocm] # Apple Silicon users (M1/M2/M3/M4 Macs) pip install simpletuner[apple] ``` For manual installation or development setup, see the [installation documentation](/documentation/INSTALL.md). ## Troubleshooting Enable debug logs for a more detailed insight by adding `export SIMPLETUNER_LOG_LEVEL=DEBUG` to your environment (`config/config.env`) file. For performance analysis of the training loop, setting `SIMPLETUNER_TRAINING_LOOP_LOG_LEVEL=DEBUG` will have timestamps that highlight any issues in your configuration. For a comprehensive list of options available, consult [this documentation](/documentation/OPTIONS.md). ", Assign "at most 3 tags" to the expected json: {"id":"11388","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"