base on Official code for the paper "LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes". <p align="center"> <img src="assets/logo_color.png" height=180> </p> # 😴 LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes 😴 <div align="center"> [](https://luciddreamer-cvlab.github.io/) [](https://arxiv.org/abs/2311.13384) [](https://colab.research.google.com/github/camenduru/LucidDreamer-Gaussian-colab/blob/main/LucidDreamer_Gaussian_colab.ipynb) [](https://huggingface.co/spaces/ironjr/LucidDreamer-mini) </div> <div align="center"> [](https://github.com/luciddreamer-cvlab/LucidDreamer) [](https://twitter.com/_ironjr_) [](https://github.com/luciddreamer-cvlab/LucidDreamer/blob/master/LICENSE) </div> https://github.com/luciddreamer-cvlab/LucidDreamer/assets/12259041/35004aaa-dffc-4133-b15a-05224e68b91e > #### [LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes](https://arxiv.org/abs/2311.13384) > ##### \*[Jaeyoung Chung](https://robot0321.github.io/), \*[Suyoung Lee](https://esw0116.github.io/), [Hyeongjin Nam](https://hygenie1228.github.io/), [Jaerin Lee](http://jaerinlee.com/), [Kyoung Mu Lee](https://cv.snu.ac.kr/index.php/~kmlee/) ###### \*Denotes equal contribution. <p align="center"> <img src="assets/logo_cvlab.png" height=60> </p> --- ## 🤖 Install ### Ubuntu #### Prerequisite - CUDA>=11.4 (higher version is OK). - Python==3.9 (cannot use 3.10 due to open3d compatibility) #### Installation script ```bash conda create -n lucid python=3.9 conda activate lucid pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d torch==2.0.1 torchvision==0.15.2 gradio omegaconf # ZoeDepth pip install timm==0.6.7 # Gaussian splatting pip install plyfile==0.8.1 cd submodules/depth-diff-gaussian-rasterization-min # sudo apt-get install libglm-dev # may be required for the compilation. python setup.py install cd ../simple-knn python setup.py install cd ../.. ``` ### Windows (Experimental, Tested on Windows 11 with VS2022) #### Checklist - Make sure that the versions of your installed [**CUDA**](https://developer.nvidia.com/cuda-11-8-0-download-archive), [**cudatoolkit**](https://anaconda.org/nvidia/cudatoolkit), and [**pytorch**](https://pytorch.org/get-started/previous-versions/) match. We have tested on CUDA==11.8. - Make sure you download and install C++ (>=14) from the [Visual Studio build tools](https://visualstudio.microsoft.com/downloads/). #### Installation script ```bash conda create -n lucid python=3.9 conda activate lucid conda install pytorch=2.0.1 torchvision=0.15.2 torchaudio=2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d gradio omegaconf # ZoeDepth pip install timm==0.6.7 # Gaussian splatting pip install plyfile==0.8.1 # There is an issue with whl file so please manually install the module now. cd submodules\depth-diff-gaussian-rasterization-min\third_party git clone https://github.com/g-truc/glm.git cd ..\ python setup.py install cd ..\simple-knn python setup.py install cd ..\.. ``` ## ⚡ Usage We offer several ways to interact with LucidDreamer: 1. A demo is available on [`ironjr/LucidDreamer` HuggingFace Space](https://huggingface.co/spaces/ironjr/LucidDreamer) (including custom SD ckpt) and [`ironjr/LucidDreamer-mini` HuggingFace Space](https://huggingface.co/spaces/ironjr/LucidDreamer-mini) (minimal features / try at here in case of the former is down) (We appreciate all the HF / Gradio team for their support). https://github.com/luciddreamer-cvlab/LucidDreamer/assets/12259041/745bfc46-8215-4db2-80d5-4825e91316bc 2. Another demo is available on a [Colab](https://colab.research.google.com/github/camenduru/LucidDreamer-Gaussian-colab/blob/main/LucidDreamer_Gaussian_colab.ipynb), implemented by [@camenduru](https://github.com/camenduru) (We greatly thank [@camenduru](https://github.com/camenduru) for the contribution). 3. You can use the gradio demo locally by running [`CUDA_VISIBLE_DEVICES=0 python app.py`](app.py) (full feature including huggingface model download, requires ~15GB) or [`CUDA_VISIBLE_DEVICES=0 python app_mini.py`](app_mini.py) (minimum viable demo, uses only SD1.5). 4. You can also run this with command line interface as described below. ### Run with your own samples ```bash # Default Example python run.py --image <path_to_image> --text <path_to_text_file> [Other options] ``` - Replace <path_to_image> and <path_to_text_file> with the paths to your image and text files. #### Other options - `--image` (`-img`): Specify the path to the input image for scene generation. - `--text` (`-t`): Path to the text file containing the prompt that guides the scene generation. - `--neg_text` (`-nt`): Optional. A negative text prompt to refine and constrain the scene generation. - `--campath_gen` (`-cg`): Choose a camera path for scene generation (options: `lookdown`, `lookaround`, `rotate360`). - `--campath_render` (`-cr`): Select a camera path for video rendering (options: `back_and_forth`, `llff`, `headbanging`). - `--model_name`: Optional. Name of the inpainting model used for dreaming. Leave blank for default(SD 1.5). - `--seed`: Set a seed value for reproducibility in the inpainting process. - `--diff_steps`: Number of steps to perform in the inpainting process. - `--save_dir` (`-s`): Directory to save the generated scenes and videos. Specify to organize outputs. ### Guideline for the prompting / Troubleshoot #### General guides 1. If your image is indoors with specific scene (and possible character in it), you can **just put the most simplest representation of the scene first**, like a cozy livingroom for christmas, or a dark garage, etc. Please avoid prompts like 1girl because it will generate many humans for each inpainting task. 2. If you want to start from already hard-engineered image from e.g., StableDiffusion model, or a photo taken from other sources, you can try **using [WD14 tagger](https://github.com/toriato/stable-diffusion-webui-wd14-tagger) ([huggingface demo](https://huggingface.co/spaces/deepghs/wd14_tagging_online)) to extract the danbooru tags from an image**. Please ensure you remove some comma separated tags if you don't want them to appear multiple times. This include human-related objects, e.g., 1girl, white shirt, boots, smiling face, red eyes, etc. Make sure to specify the objects you want to have multiples of them. #### Q. I generate unwanted objects everywhere, e.g., photo frames. 1. Manipulate negative prompts to set harder constraints for the frame object. You may try adding tags like twitter thumbnail, profile image, instagram image, watermark, text to the negative prompt. In fact, negative prompts are the best thing to try if you want some things not to be appeared in the resulting image. 2. Try using other custom checkpoint models, which employs different pipeline methods: LaMa inpainting -> ControlNet-inpaint guided image inpainting. ### Visualize `.ply` files There are multiple available viewers / editors for Gaussian splatting `.ply` files. 1. [@playcanvas](https://github.com/playcanvas)'s [Super-Splat](https://github.com/playcanvas/super-splat) project ([Live demo](https://playcanvas.com/super-splat)). This is the viewer we have used for our debugging along with MeshLab.  2. [@antimatter15](https://github.com/antimatter15)'s [WebGL viewer](https://github.com/antimatter15/splat) for Gaussian splatting ([Live demo](https://antimatter15.com/splat/)). 3. [@splinetool](https://github.com/splinetool)'s [web-based viewer](https://spline.design/) for Gaussian splatting. This is the version we have used in our project page's demo. ## 🚩 **Updates** - ✅ December 12, 2023: We have precompiled wheels for the CUDA-based submodules and put them in `submodules/wheels`. The Windows installation guide is revised accordingly! - ✅ December 11, 2023: We have updated installation guides for Windows. Thank you [@Maoku](https://twitter.com/Maoku) for your great contribution! - ✅ December 8, 2023: [HuggingFace Space demo](https://huggingface.co/spaces/ironjr/LucidDreamer) is out. We deeply thank all the HF team for their support! - ✅ December 7, 2023: [Colab](https://colab.research.google.com/github/camenduru/LucidDreamer-Gaussian-colab/blob/main/LucidDreamer_Gaussian_colab.ipynb) implementation is now available thanks to [@camenduru](https://github.com/camenduru)! - ✅ December 6, 2023: Code release! - ✅ November 22, 2023: We have released our paper, LucidDreamer on [arXiv](https://arxiv.org/abs/2311.13384). ## 🌏 Citation Please cite us if you find our project useful! ```latex @article{chung2023luciddreamer, title={LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes}, author={Chung, Jaeyoung and Lee, Suyoung and Nam, Hyeongjin and Lee, Jaerin and Lee, Kyoung Mu}, journal={arXiv preprint arXiv:2311.13384}, year={2023} } ``` ## 🤗 Acknowledgement We deeply appreciate [ZoeDepth](https://github.com/isl-org/ZoeDepth), [Stability AI](), and [Runway](https://huggingface.co/runwayml/stable-diffusion-v1-5) for their models. ## 📧 Contact If you have any questions, please email `[email protected]`, `[email protected]`, `[email protected]`. ## ⭐ Star History <a href="https://star-history.com/#luciddreamer-cvlab/LucidDreamer&Date"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=luciddreamer-cvlab/LucidDreamer&type=Date&theme=dark" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=luciddreamer-cvlab/LucidDreamer&type=Date" /> <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=luciddreamer-cvlab/LucidDreamer&type=Date" /> </picture> </a> ", Assign "at most 3 tags" to the expected json: {"id":"5784","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"