base on VideoSys: An easy and efficient system for video generation <p align="center"> <img width="55%" alt="VideoSys" src="./assets/figures/logo.png?raw=true"> </p> <h3 align="center"> An easy and efficient system for video generation </h3> <p align="center">| <a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#installation">Quick Start</a> | <a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#usage">Supported Models</a> | <a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#acceleration-techniques">Accelerations</a> | <a href="https://discord.gg/WhPmYm9FeG">Discord</a> | <a href="https://oahzxl.notion.site/VideoSys-News-42391db7e0a44f96a1f0c341450ae472?pvs=4">Media</a> | <a href="https://huggingface.co/VideoSys">HuggingFace Space</a> | </p> ### Latest News 🔥 - [2024/11] 🔥 <b>Propose Data-Centric Parallel (DCP) [[blog](https://oahzxl.github.io/DCP/)][[doc](./docs/dcp.md)], a simple and efficient method for variable sequences (<i>e.g., </i> videos) training</b>. - [2024/09] Support [CogVideoX](https://github.com/THUDM/CogVideo), [Vchitect-2.0](https://github.com/Vchitect/Vchitect-2.0) and [Open-Sora-Plan v1.2.0](https://github.com/PKU-YuanGroup/Open-Sora-Plan). - [2024/08] 🔥 Evole from [OpenDiT](https://github.com/NUS-HPC-AI-Lab/VideoSys/tree/v1.0.0) to <b>VideoSys: An easy and efficient system for video generation</b>. - [2024/08] 🔥 Release PAB paper: <b>[Real-Time Video Generation with Pyramid Attention Broadcast](https://arxiv.org/abs/2408.12588)</b>. - [2024/06] 🔥 Propose Pyramid Attention Broadcast (PAB) [[paper](https://arxiv.org/abs/2408.12588)][[blog](https://oahzxl.github.io/PAB/)][[doc](./docs/pab.md)], the first approach to achieve <b>real-time</b> DiT-based video generation, delivering <b>negligible quality loss</b> without <b>requiring any training</b>. - [2024/06] Support [Open-Sora-Plan](https://github.com/PKU-YuanGroup/Open-Sora-Plan) and [Latte](https://github.com/Vchitect/Latte). - [2024/03] 🔥 Propose Dynamic Sequence Parallel (DSP)[[paper](https://arxiv.org/abs/2403.10266)][[doc](./docs/dsp.md)], achieves **3x** speed for training and **2x** speed for inference in Open-Sora compared with sota sequence parallelism. - [2024/03] Support [Open-Sora](https://github.com/hpcaitech/Open-Sora). - [2024/02] 🎉 Release [OpenDiT](https://github.com/NUS-HPC-AI-Lab/VideoSys/tree/v1.0.0): An Easy, Fast and Memory-Efficent System for DiT Training and Inference. # About VideoSys is an open-source project that provides a user-friendly and high-performance infrastructure for video generation. This comprehensive toolkit will support the entire pipeline from training and inference to serving and compression. We are committed to continually integrating cutting-edge open-source video models and techniques. Stay tuned for exciting enhancements and new features on the horizon! ## Installation Prerequisites: - Python >= 3.10 - PyTorch >= 1.13 (We recommend to use a >2.0 version) - CUDA >= 11.6 We strongly recommend using Anaconda to create a new environment (Python >= 3.10) to run our examples: ```shell conda create -n videosys python=3.10 -y conda activate videosys ``` Install VideoSys: ```shell git clone https://github.com/NUS-HPC-AI-Lab/VideoSys cd VideoSys pip install -e . ``` ## Usage VideoSys supports many diffusion models with our various acceleration techniques, enabling these models to run faster and consume less memory. <b>You can find all available models and their supported acceleration techniques in the following table. Click `Code` to see how to use them.</b> <table> <tr> <th rowspan="2">Model</th> <th rowspan="2">Train</th> <th rowspan="2">Infer</th> <th colspan="3">Acceleration Techniques</th> </tr> <tr> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#dyanmic-sequence-parallelism-dsp-paperdoc">DSP</a></th> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#pyramid-attention-broadcast-pab-blogdoc">PAB</a></th> <th><a href="https://github.com/NUS-HPC-AI-Lab/VideoSys?tab=readme-ov-file#pyramid-attention-broadcast-pab-blogdoc">DCP</a></th> </tr> <tr> <td>Vchitect [<a href="https://github.com/Vchitect/Vchitect-2.0">source</a>]</td> <td align="center">/</td> <td align="center"><a href="./examples/inference/vchitect/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">/</td> </tr> <tr> <td>CogVideoX [<a href="https://github.com/THUDM/CogVideo">source</a>]</td> <td align="center">🟡</td> <td align="center"><a href="./examples/inference/cogvideox/sample.py">Code</a></td> <td align="center">/</td> <td align="center">✅</td> <td align="center">🟡</td> </tr> <tr> <td>Latte [<a href="https://github.com/Vchitect/Latte">source</a>]</td> <td align="center">/</td> <td align="center"><a href="./examples/inference/latte/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">/</td> </tr> <tr> <td>Open-Sora-Plan [<a href="https://github.com/PKU-YuanGroup/Open-Sora-Plan">source</a>]</td> <td align="center">/</td> <td align="center"><a href="./examples/inference/open_sora_plan/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">/</td> </tr> <tr> <td>Open-Sora [<a href="https://github.com/hpcaitech/Open-Sora">source</a>]</td> <td align="center"><a href="./examples/training/open_sora">Code</a></td> <td align="center"><a href="./examples/inference/open_sora/sample.py">Code</a></td> <td align="center">✅</td> <td align="center">✅</td> <td align="center">✅</td> </tr> </table> You can also find easy demo with HuggingFace Space <a href="https://huggingface.co/VideoSys">[link]</a> and Gradio <a href="./gradio">[link]</a>. 🟡 means work in progress. ## Acceleration Techniques ### Data-Centric Parallel (DCP) [[blog](https://oahzxl.github.io/DCP/)][[doc](./docs/dcp.md)] <!--  --> <p align="center"> <img src="./assets/figures/dcp_overview.png" alt="method" height="300"> </p> Data-Centric Parallel (DCP) is a simple but effective approach to accelerate distributed training of variable sequences. Unlike previous methods that fix training settings, DCP dyanmically adjusts parallelism and other configs driven by incoming data during runtime, achieving up to 2.1x speedup. As a ease-of-use method, DCP can enpower any video models and parallel methods with minimal code changes. See its details [here](./docs/dcp.md). ---- ### Pyramid Attention Broadcast (PAB) [[paper](https://arxiv.org/abs/2408.12588)][[blog](https://oahzxl.github.io/PAB/)][[doc](./docs/pab.md)]  PAB is the first approach to achieve <b>real-time</b> DiT-based video generation, delivering <b>lossless quality</b> without <b>requiring any training</b>. By mitigating redundant attention computation, PAB achieves up to 21.6 FPS with 10.6x acceleration, without sacrificing quality across popular DiT-based video generation models including [Open-Sora](https://github.com/hpcaitech/Open-Sora), [Latte](https://github.com/Vchitect/Latte) and [Open-Sora-Plan](https://github.com/PKU-YuanGroup/Open-Sora-Plan). See its details [here](./docs/pab.md). ---- ### Dyanmic Sequence Parallelism (DSP) [[paper](https://arxiv.org/abs/2403.10266)][[doc](./docs/dsp.md)]  DSP is a novel, elegant and super efficient sequence parallelism for [Open-Sora](https://github.com/hpcaitech/Open-Sora), [Latte](https://github.com/Vchitect/Latte) and other multi-dimensional transformer architecture. It achieves **3x** speed for training and **2x** speed for inference in Open-Sora compared with sota sequence parallelism ([DeepSpeed Ulysses](https://arxiv.org/abs/2309.14509)). For a 10s (80 frames) of 512x512 video, the inference latency of Open-Sora is: | Method | 1xH800 | 8xH800 (DS Ulysses) | 8xH800 (DSP) | | ------ | ------ | ------ | ------ | | Latency(s) | 106 | 45 | 22 | See its details [here](./docs/dsp.md). ## Contributing We welcome and value any contributions and collaborations. Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved. ## Contributors <a href="https://github.com/NUS-HPC-AI-Lab/VideoSys/graphs/contributors"> <img src="https://contrib.rocks/image?repo=NUS-HPC-AI-Lab/VideoSys"/> </a> ## Star History [](https://star-history.com/#NUS-HPC-AI-Lab/VideoSys&Date) ## Citation ``` @misc{videosys2024, author={VideoSys Team}, title={VideoSys: An Easy and Efficient System for Video Generation}, year={2024}, publisher={GitHub}, url = {https://github.com/NUS-HPC-AI-Lab/VideoSys}, } @misc{zhao2024pab, title={Real-Time Video Generation with Pyramid Attention Broadcast}, author={Xuanlei Zhao and Xiaolong Jin and Kai Wang and Yang You}, year={2024}, eprint={2408.12588}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2408.12588}, } @misc{zhao2024dsp, title={DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers}, author={Xuanlei Zhao and Shenggan Cheng and Chang Chen and Zangwei Zheng and Ziming Liu and Zheming Yang and Yang You}, year={2024}, eprint={2403.10266}, archivePrefix={arXiv}, primaryClass={cs.DC}, url={https://arxiv.org/abs/2403.10266}, } @misc{zhao2024opendit, author={Xuanlei Zhao, Zhongkai Zhao, Ziming Liu, Haotian Zhou, Qianli Ma, and Yang You}, title={OpenDiT: An Easy, Fast and Memory-Efficient System for DiT Training and Inference}, year={2024}, publisher={GitHub}, url={https://github.com/NUS-HPC-AI-Lab/VideoSys/tree/v1.0.0}, } ``` ", Assign "at most 3 tags" to the expected json: {"id":"11590","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"