base on [CVPR 2024 - Oral, Best Paper Award Candidate] Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation # Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation **CVPR 2024 (Oral, Best Paper Award Candidate)** This repository represents the official implementation of the paper titled "Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation". [](https://marigoldmonodepth.github.io) [](https://arxiv.org/abs/2312.02145) [-Space-yellow)](https://huggingface.co/spaces/prs-eth/marigold-lcm) [-Model-green)](https://huggingface.co/prs-eth/marigold-lcm-v1-0) [](https://colab.research.google.com/drive/12G8reD13DdpMie5ZQlaFNo2WCGeNUH-u?usp=sharing) [](https://www.apache.org/licenses/LICENSE-2.0) <!-- [](https://huggingface.co/prs-eth/marigold-v1-0) --> <!-- [](https://arxiv.org/abs/2312.02145) --> <!-- [](https://github.com/prs-eth/Marigold) --> <!-- []() --> <!-- []() --> [Bingxin Ke](http://www.kebingxin.com/), [Anton Obukhov](https://www.obukhov.ai/), [Shengyu Huang](https://shengyuh.github.io/), [Nando Metzger](https://nandometzger.github.io/), [Rodrigo Caye Daudt](https://rcdaudt.github.io/), [Konrad Schindler](https://scholar.google.com/citations?user=FZuNgqIAAAAJ&hl=en ) We present Marigold, a diffusion model, and associated fine-tuning protocol for monocular depth estimation. Its core principle is to leverage the rich visual knowledge stored in modern generative image models. Our model, derived from Stable Diffusion and fine-tuned with synthetic data, can zero-shot transfer to unseen data, offering state-of-the-art monocular depth estimation results.  ## 📢 News 2024-05-28: Training code is released.<br> 2024-03-23: Added [LCM v1.0](https://huggingface.co/prs-eth/marigold-lcm-v1-0) for faster inference - try it out at <a href="https://huggingface.co/spaces/prs-eth/marigold-lcm"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face%20(LCM)-Space-yellow" height="16"></a><br> 2024-03-04: Accepted to CVPR 2024. <br> 2023-12-22: Contributed to Diffusers [community pipeline](https://github.com/huggingface/diffusers/tree/main/examples/community#marigold-depth-estimation). <br> 2023-12-19: Updated [license](LICENSE.txt) to Apache License, Version 2.0.<br> 2023-12-08: Added <a href="https://huggingface.co/spaces/toshas/marigold"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-Space-yellow" height="16"></a> - try it out with your images for free!<br> 2023-12-05: Added <a href="https://colab.research.google.com/drive/12G8reD13DdpMie5ZQlaFNo2WCGeNUH-u?usp=sharing"><img src="doc/badges/badge-colab.svg" height="16"></a> - dive deeper into our inference pipeline!<br> 2023-12-04: Added <a href="https://arxiv.org/abs/2312.02145"><img src="https://img.shields.io/badge/arXiv-PDF-b31b1b" height="16"></a> paper and inference code (this repository). ## 🚀 Usage **We offer several ways to interact with Marigold**: 1. We integrated [Marigold Pipelines into diffusers 🧨](https://huggingface.co/docs/diffusers/api/pipelines/marigold). Check out many exciting usage scenarios in [this diffusers tutorial](https://huggingface.co/docs/diffusers/using-diffusers/marigold_usage). 1. A free online interactive demo is available here: <a href="https://huggingface.co/spaces/prs-eth/marigold-lcm"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face%20(LCM)-Space-yellow" height="16"></a> (kudos to the HF team for the GPU grant) 1. Run the demo locally (requires a GPU and an `nvidia-docker2`, see [Installation Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)): 1. Paper version: `docker run -it -p 7860:7860 --platform=linux/amd64 --gpus all registry.hf.space/toshas-marigold:latest python app.py` 1. LCM version: `docker run -it -p 7860:7860 --platform=linux/amd64 --gpus all registry.hf.space/prs-eth-marigold-lcm:latest python app.py` 1. Extended demo on a Google Colab: <a href="https://colab.research.google.com/drive/12G8reD13DdpMie5ZQlaFNo2WCGeNUH-u?usp=sharing"><img src="doc/badges/badge-colab.svg" height="16"></a> 1. If you just want to see the examples, visit our gallery: <a href="https://marigoldmonodepth.github.io"><img src="doc/badges/badge-website.svg" height="16"></a> 1. Finally, local development instructions with this codebase are given below. ## 🛠️ Setup The inference code was tested on: - Ubuntu 22.04 LTS, Python 3.10.12, CUDA 11.7, GeForce RTX 3090 (pip) ### 🪧 A Note for Windows users We recommend running the code in WSL2: 1. Install WSL following [installation guide](https://learn.microsoft.com/en-us/windows/wsl/install#install-wsl-command). 1. Install CUDA support for WSL following [installation guide](https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-support-for-wsl-2). 1. Find your drives in `/mnt/<drive letter>/`; check [WSL FAQ](https://learn.microsoft.com/en-us/windows/wsl/faq#how-do-i-access-my-c--drive-) for more details. Navigate to the working directory of choice. ### 📦 Repository Clone the repository (requires git): ```bash git clone https://github.com/prs-eth/Marigold.git cd Marigold ``` ### 💻 Dependencies We provide several ways to install the dependencies. 1. **Using [Mamba](https://github.com/mamba-org/mamba)**, which can installed together with [Miniforge3](https://github.com/conda-forge/miniforge?tab=readme-ov-file#miniforge3). Windows users: Install the Linux version into the WSL. After the installation, Miniforge needs to be activated first: `source /home/$USER/miniforge3/bin/activate`. Create the environment and install dependencies into it: ```bash mamba env create -n marigold --file environment.yaml conda activate marigold ``` 2. **Using pip:** Alternatively, create a Python native virtual environment and install dependencies into it: ```bash python -m venv venv/marigold source venv/marigold/bin/activate pip install -r requirements.txt ``` Keep the environment activated before running the inference script. Activate the environment again after restarting the terminal session. ## 🏃 Testing on your images ### 📷 Prepare images 1. Use selected images from our paper: ```bash bash script/download_sample_data.sh ``` 1. Or place your images in a directory, for example, under `input/in-the-wild_example`, and run the following inference command. ### 🚀 Run inference with LCM (faster) The [LCM checkpoint](https://huggingface.co/prs-eth/marigold-lcm-v1-0) is distilled from our original checkpoint towards faster inference speed (by reducing inference steps). The inference steps can be as few as 1 (default) to 4. Run with default LCM setting: ```bash python run.py \ --input_rgb_dir input/in-the-wild_example \ --output_dir output/in-the-wild_example_lcm ``` ### 🎮 Run inference with DDIM (paper setting) This setting corresponds to our paper. For academic comparison, please run with this setting. ```bash python run.py \ --checkpoint prs-eth/marigold-v1-0 \ --denoise_steps 50 \ --ensemble_size 10 \ --input_rgb_dir input/in-the-wild_example \ --output_dir output/in-the-wild_example ``` You can find all results in `output/in-the-wild_example`. Enjoy! ### ⚙️ Inference settings The default settings are optimized for the best result. However, the behavior of the code can be customized: - Trade-offs between the **accuracy** and **speed** (for both options, larger values result in better accuracy at the cost of slower inference.) - `--ensemble_size`: Number of inference passes in the ensemble. For LCM `ensemble_size` is more important than `denoise_steps`. Default: ~~10~~ 5 (for LCM). - `--denoise_steps`: Number of denoising steps of each inference pass. For the original (DDIM) version, it's recommended to use 10-50 steps, while for LCM 1-4 steps. When unassigned (`None`), will read default setting from model config. Default: ~~10 4 (for LCM)~~ `None`. - By default, the inference script resizes input images to the *processing resolution*, and then resizes the prediction back to the original resolution. This gives the best quality, as Stable Diffusion, from which Marigold is derived, performs best at 768x768 resolution. - `--processing_res`: the processing resolution; set as 0 to process the input resolution directly. When unassigned (`None`), will read default setting from model config. Default: ~~768~~ `None`. - `--output_processing_res`: produce output at the processing resolution instead of upsampling it to the input resolution. Default: False. - `--resample_method`: the resampling method used to resize images and depth predictions. This can be one of `bilinear`, `bicubic`, or `nearest`. Default: `bilinear`. - `--half_precision` or `--fp16`: Run with half-precision (16-bit float) to have faster speed and reduced VRAM usage, but might lead to suboptimal results. - `--seed`: Random seed can be set to ensure additional reproducibility. Default: None (unseeded). Note: forcing `--batch_size 1` helps to increase reproducibility. To ensure full reproducibility, [deterministic mode](https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms) needs to be used. - `--batch_size`: Batch size of repeated inference. Default: 0 (best value determined automatically). - `--color_map`: [Colormap](https://matplotlib.org/stable/users/explain/colors/colormaps.html) used to colorize the depth prediction. Default: Spectral. Set to `None` to skip colored depth map generation. - `--apple_silicon`: Use Apple Silicon MPS acceleration. ### ⬇ Checkpoint cache By default, the [checkpoint](https://huggingface.co/prs-eth/marigold-v1-0) is stored in the Hugging Face cache. The `HF_HOME` environment variable defines its location and can be overridden, e.g.: ```bash export HF_HOME=$(pwd)/cache ``` Alternatively, use the following script to download the checkpoint weights locally: ```bash bash script/download_weights.sh marigold-v1-0 # or LCM checkpoint bash script/download_weights.sh marigold-lcm-v1-0 ``` At inference, specify the checkpoint path: ```bash python run.py \ --checkpoint checkpoint/marigold-v1-0 \ --denoise_steps 50 \ --ensemble_size 10 \ --input_rgb_dir input/in-the-wild_example\ --output_dir output/in-the-wild_example ``` ## 🦿 Evaluation on test datasets <a name="evaluation"></a> Install additional dependencies: ```bash pip install -r requirements+.txt -r requirements.txt ``` Set data directory variable (also needed in evaluation scripts) and download [evaluation datasets](https://share.phys.ethz.ch/~pf/bingkedata/marigold/evaluation_dataset) into corresponding subfolders: ```bash export BASE_DATA_DIR=<YOUR_DATA_DIR> # Set target data directory wget -r -np -nH --cut-dirs=4 -R "index.html*" -P ${BASE_DATA_DIR} https://share.phys.ethz.ch/~pf/bingkedata/marigold/evaluation_dataset/ ``` Run inference and evaluation scripts, for example: ```bash # Run inference bash script/eval/11_infer_nyu.sh # Evaluate predictions bash script/eval/12_eval_nyu.sh ``` Note: although the seed has been set, the results might still be slightly different on different hardware. ## 🏋️ Training Based on the previously created environment, install extended requirements: ```bash pip install -r requirements++.txt -r requirements+.txt -r requirements.txt ``` Set environment parameters for the data directory: ```bash export BASE_DATA_DIR=YOUR_DATA_DIR # directory of training data export BASE_CKPT_DIR=YOUR_CHECKPOINT_DIR # directory of pretrained checkpoint ``` Download Stable Diffusion v2 [checkpoint](https://huggingface.co/stabilityai/stable-diffusion-2) into `${BASE_CKPT_DIR}` Prepare for [Hypersim](https://github.com/apple/ml-hypersim) and [Virtual KITTI 2](https://europe.naverlabs.com/research/computer-vision/proxy-virtual-worlds-vkitti-2/) datasets and save into `${BASE_DATA_DIR}`. Please refer to [this README](script/dataset_preprocess/hypersim/README.md) for Hypersim preprocessing. Run training script ```bash python train.py --config config/train_marigold.yaml ``` Resume from a checkpoint, e.g. ```bash python train.py --resume_run output/marigold_base/checkpoint/latest ``` Evaluating results Only the U-Net is updated and saved during training. To use the inference pipeline with your training result, replace `unet` folder in Marigold checkpoints with that in the `checkpoint` output folder. Then refer to [this section](#evaluation) for evaluation. **Note**: Although random seeds have been set, the training result might be slightly different on different hardwares. It's recommended to train without interruption. ## ✏️ Contributing Please refer to [this](CONTRIBUTING.md) instruction. ## 🤔 Troubleshooting | Problem | Solution | |----------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------| | (Windows) Invalid DOS bash script on WSL | Run `dos2unix <script_name>` to convert script format | | (Windows) error on WSL: `Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory` | Run `export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH` | ## 🎓 Citation Please cite our paper: ```bibtex @InProceedings{ke2023repurposing, title={Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation}, author={Bingxin Ke and Anton Obukhov and Shengyu Huang and Nando Metzger and Rodrigo Caye Daudt and Konrad Schindler}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2024} } ``` ## 🎫 License This work is licensed under the Apache License, Version 2.0 (as defined in the [LICENSE](LICENSE.txt)). By downloading and using the code and model you agree to the terms in the [LICENSE](LICENSE.txt). [](https://www.apache.org/licenses/LICENSE-2.0) ", Assign "at most 3 tags" to the expected json: {"id":"5812","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"