base on # Bootstrap 3D-GS <!-- Project Contribution: [Yifei Gao](https://github.com/yileijin), [Jie Ou](https://github.com/oujieww) <br /> Our Paper: [[`arxiv`](https://arxiv.org/pdf/2404.18669)] --> ## Overview In our paper, we present a bootstrapping method that significantly addresses this problem. This approach employs a diffusion model to enhance the rendering of novel views using trained 3D-GS, thereby streamlining the training process. Our results indicate that bootstrapping effectively reduces artifacts, as well as clear enhancements on the evaluation metrics. Furthermore, we show that our method is versatile and can be easily integrated, allowing various 3D reconstruction projects to benefit from our approach. <p align="center"> <img src="assets/overall_comparison.png" width=100% height=100% class="center"> </p> ## Installation Our program is tested on cuda 11.8 with pytorch==2.0.1, this environment is suitable for both 3D-GS and the stable-diffusion (xformer) 1. Clone this repo: ``` git clone https://github.com/yileijin/Bootstrap-3D-GS.git --recursive ``` 2. Install dependencies ``` SET DISTUTILS_USE_SDK=1 # Windows only conda create -n boot_3dgs python==3.11.8 conda activate boot_3dgs pip install -r requirements.txt ``` ## Data First, create a ```data/``` folder inside the project path by ``` mkdir data ``` The data structure will be organised as follows: ``` data/ ├── dataset_name │ ├── scene1/ │ │ ├── images │ │ │ ├── IMG_0.jpg │ │ │ ├── IMG_1.jpg │ │ │ ├── ... │ │ ├── sparse/ │ │ └──0/ │ ├── scene2/ │ │ ├── images │ │ │ ├── IMG_0.jpg │ │ │ ├── IMG_1.jpg │ │ │ ├── ... │ │ ├── sparse/ │ │ └──0/ ... ``` ### Stable Diffusion CKPT The stable diffusion model we use can be downloaded in [SD2.1v](https://huggingface.co/stabilityai/stable-diffusion-2-1)/[x4-upscaling](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler) ### Public Data The MipNeRF360 scenes are provided by the paper author [here](https://jonbarron.info/mipnerf360/). And we test on scenes ```bicycle, bonsai, counter, garden, kitchen, room, stump```. The SfM data sets for Tanks&Temples and Deep Blending are hosted by 3D-Gaussian-Splatting [here](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip). Download and uncompress them into the ```data/``` folder. ## Training Our training process is the same as the original [3D-GS](https://github.com/graphdeco-inria/gaussian-splatting), but regarding the Diffusion Model part, we strongly recommend that you make modifications yourself in the configuration ```./arguments ```. Some important configurations are detailed in our paper. For example, to train a scene: ``` python train.py -s <path to COLMAP or NeRF Synthetic dataset> ``` ## Novel-view Rendering Our rendering also includes scenes that are not in the training set or the test set. By setting ```do_expension=True```, you can obtain completely novel views. ``` python render.py -m <path to trained model> # Generate renderings python metrics.py -m <path to trained model> # Compute error metrics on renderings ``` <!-- ## Contact - Yifei Gao: [email protected] ## Citation If you find our work helpful, please consider citing: ```bibtex @misc{gao2024bootstrap, title={Bootstrap 3D Reconstructed Scenes from 3D Gaussian Splatting}, author={Yifei Gao and Jie Ou and Lei Wang and Jun Cheng}, year={2024}, eprint={2404.18669}, archivePrefix={arXiv}, primaryClass={cs.GR} } ``` --> ## LICENSE Please follow the LICENSE of [3D-GS](https://github.com/graphdeco-inria/gaussian-splatting). ## Acknowledgement We thank all authors from [3D-GS](https://github.com/graphdeco-inria/gaussian-splatting) for presenting such an excellent work. ", Assign "at most 3 tags" to the expected json: {"id":"10263","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"