base on [ECCV 2024] Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting # Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting ### [Project Page](https://zju3dv.github.io/street_gaussians) | [Paper](https://arxiv.org/pdf/2401.01339.pdf) | [Unofficial Implementation](https://github.com/LightwheelAI/street-gaussians-ns) > [Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting](https://arxiv.org/abs/2401.01339) > Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng > ECCV 2024 https://github.com/user-attachments/assets/f28a64bd-9932-4447-b710-9254ae5ed56f ### Installation <details> <summary>Clone this repository</summary> ``` git clone https://github.com/zju3dv/street_gaussians.git ``` </details> <details> <summary>Set up the python environment</summary> ``` # Set conda environment conda create -n street-gaussian python=3.8 conda activate street-gaussian # Install torch (corresponding to your CUDA version) pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 # Install requirements pip install -r requirements.txt # Install submodules pip install ./submodules/diff-gaussian-rasterization pip install ./submodules/simple-knn pip install ./submodules/simple-waymo-open-dataset-reader python script/test_gaussian_rasterization.py ``` </details> <details> <summary>Prepare Waymo Open Dataset.</summary> We provide the example scenes [here](https://drive.google.com/drive/folders/1ghpE_kBwqXiWgiSWAajByjPsmj1y0l1H). You can directly download the data and skip the following steps for a quick start. #### Download the training and validation set of [Waymo Open Dataset](https://console.cloud.google.com/storage/browser/waymo_open_dataset_v_1_4_1/individual_files?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))). We provide the split file following [EmerNeRF](https://emernerf.github.io/https://emernerf.github.io/). You can refer to [this document](https://github.com/NVlabs/EmerNeRF/blob/main/docs/NOTR.md) for download details. <!-- Please note that `val_dynamic.txt` specify scenes from the validation set, which means you may need to change the file source [here](https://github.com/NVlabs/EmerNeRF/blob/8c051d7cccbad3b52c7b11a519c971b8ead97e1a/datasets/download_waymo.py#L31). --> #### Preprocess the data Download the tracking predictions on validation set, We provide the processed results [here](https://drive.google.com/file/d/1bMDOMtZdyP3m8qY1Phb5Sr6Po-QWFIWk/view?usp=drive_link). Preprocess the example scenes ``` python script/waymo/waymo_converter.py --root_dir TRAINING_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/demo.txt --segment_file script/waymo/waymo_splits/segment_list_train.txt ``` Preprocess the experiment scenes ``` python script/waymo/waymo_converter.py --root_dir VALIDATION_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/val_dynamic.txt --segment_file script/waymo/waymo_splits/segment_list_val.txt --track_file TRACKER_PATH ``` Generating LiDAR depth ``` python script/waymo/generate_lidar_depth.py --datadir DATA_DIR ``` Generating sky mask Install GroundingDINO following [this repo](https://github.com/IDEA-Research/GroundingDINO) and download SAM checkpoint from [this link](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth). ``` python script/waymo/generate_sky_mask.py --datadir DATA_DIR --sam_checkpoint SAM_CKPT ``` </details> <details> <summary>Prepare Custom Dataset.</summary> TODO </details> ### Configuration We build the configuration based on [3D Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting/blob/main/arguments/__init__.py). The parameters used are listed in `config.py` as shown [here](https://github.com/zju3dv/street_gaussians/blob/main/lib/config/config.py) with brief comments. - For monocular video reconstruction as demonstrated in the paper, it is recommended to use configuration provided in the [experiments](https://github.com/zju3dv/street_gaussians/tree/main/configs/experiments_waymo) directory. - For reconstruction of general sequences, it is recommenned to use `default.yaml` as shown [here](https://github.com/zju3dv/street_gaussians/blob/main/configs/default.yaml). You may need to adjust certain parameters to suit your specific sequences. ### Training ``` python train.py --config configs/xxxx.yaml ``` Training on example scenes ``` bash script/waymo/train_waymo_expample.sh ``` Training on experiment scenes ``` bash script/waymo/train_waymo_exp.sh ``` ### Rendering ``` python render.py --config configs/xxxx.yaml mode {evaluate, trajectory} ``` Rendering on example scenes ``` bash script/waymo/render_waymo_expample.sh ``` Rendering on experiment scenes ``` bash script/waymo/render_waymo_exp.sh ``` ### Visualization You can convert the scene at one certain frame into the format that can be viewed in [SIBR_viewers](https://gitlab.inria.fr/sibr/sibr_core). ``` python make_ply.py --config configs/xxxx.yaml viewer.frame_id {frame_idx} mode evaluate ``` ### Pipeline  ### Citation If you find this code useful for your research, please use the following BibTeX entry. ``` @inproceedings{yan2024street, title={Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting}, author={Yunzhi Yan and Haotong Lin and Chenxu Zhou and Weijie Wang and Haiyang Sun and Kun Zhan and Xianpeng Lang and Xiaowei Zhou and Sida Peng}, booktitle={ECCV}, year={2024} } ``` ", Assign "at most 3 tags" to the expected json: {"id":"6657","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"