base on [3DV 2025] Code for "FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent" by Cameron Smith*, David Charatan*, Ayush Tewari, and Vincent Sitzmann # FlowMap https://github.com/dcharatan/flowmap/assets/13124225/9dc9cc9a-083e-4fd1-b833-09365385cf59 This is the official implementation for **FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent** by Cameron Smith*, David Charatan*, Ayush Tewari, and Vincent Sitzmann. Check out the project website [here](https://cameronosmith.github.io/flowmap/). ## Installation To get started on Linux, create a Python virtual environment: ```bash python3.11 -m venv venv source venv/bin/activate pip install -r requirements.txt ``` For pretraining, make sure GMFlow is installed as a submodule: ``` git submodule update --init --recursive ``` If the above requirements don't work, you can try `requirements_exact.txt` instead. ## Running the Code The main entry point is `flowmap/overfit.py`. Call it via: ```bash python3 -m flowmap.overfit dataset=images dataset.images.root=path/to/folder/with/images ``` Make sure the virtual environment has been activated via `source venv/bin/activate` first. ## Pre-trained Initialization The checkpoint we used to initialize FlowMap can be found [here](https://drive.google.com/drive/folders/1PqByQSfzyLjfdZZDwn6RXIECso7WB9IY?usp=drive_link). To train your own, download the [Real Estate 10k](https://google.github.io/realestate10k/) and [CO3Dv2](https://github.com/facebookresearch/co3d) datasets and run the following script: ```bash python3 -m flowmap.pretrain ``` Some of the videos in the Real Estate 10k dataset are no longer publicly available. Reach out to us via email if you want our downloaded version of the dataset. ## Evaluation Datasets We evaluated FlowMap using video subsets of the [Local Light Field Fusion (LLFF)](https://drive.google.com/drive/folders/1M-_Fdn4ajDa0CS8-iqejv0fQQeuonpKF?usp=drive_link), [Mip-NeRF 360](https://jonbarron.info/mipnerf360/), and [Tanks & Temples](https://www.tanksandtemples.org/download/) datasets. We've uploaded a compilation of these datasets [here](https://drive.google.com/drive/folders/1PqByQSfzyLjfdZZDwn6RXIECso7WB9IY?usp=drive_link). <details> <summary>Dataset Details</summary> ### NeRF Local Light Field Fusion (LLFF) Scenes These are the LLFF scenes from the [NeRF](https://www.matthewtancik.com/nerf) paper, which were originally uploaded [here](https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7?usp=drive_link). We used all 8 scenes (`fern`, `flower`, `fortress`, `horns`, `leaves`, `orchids`, `room`, and `trex`). ### Mip-NeRF 360 Scenes These are scenes from the [Mip-NeRF 360](https://jonbarron.info/mipnerf360/) paper, which were originally uploaded [here](http://storage.googleapis.com/gresearch/refraw360/360_v2.zip). We used the `bonsai`, `counter`, and `kitchen` scenes. The original `kitchen` scene consists of several concatenated video sequences; for FlowMap, we use the first one (65 frames). We also included the `garden` scene, which is somewhat video-like, but contain large jumps that make optical flow estimation struggle. ### Tanks & Temples Scenes We used all scenes from the [Tanks & Temples](https://tanksandtemples.org/download/) dataset: `auditorium`, `ballroom`, `barn`, `caterpillar`, `church`, `courthouse`, `family`, `francis`, `horse`, `ignatius`, `lighthouse`, `m60`, `meetingroom`, `museum`, `palace`, `panther`, `playground`, `temple`, `train`, and `truck`. We preprocessed the raw videos from the dataset using the script at `flowmap/subsample.py`. This script samples 150 frames from the first minute of video evenly based on mean optical flow. </details> ## Running Ablations Each ablation shown in the paper has a [Hydra](https://hydra.cc/docs/intro/) configuration at `config/experiment`. For example, to run the ablation where point tracking is disabled, add `+experiment=ablation_no_tracks` to the overfitting command. Note that you can stack most of the ablations, e.g., `+experiment=[ablation_no_tracks,ablation_random_initialization]`. ## Generating Novel View Synthesis Results We used a modified version of the original 3D Gaussian Splatting code that backpropagates into camera positions in order to generate the novel view synthesis results shown in the paper. You can find it [here](https://github.com/cameronosmith/gaussian_barf). ## Figure and Table Generation Some of the code used to generate the tables and figures in the paper can be found in the `assets` folder. We used this code alongside Figma and LaTeXiT to create the figures in the paper. You can find our Figma file [here](https://www.figma.com/file/WLHx9d6qDRol9xowBTnHf7/FlowMap-Figures?type=design&node-id=0-1&mode=design). See `.vscode/launch.json` for the commands needed to run figure generation. ## BibTeX ```bibtex @inproceedings{smith24flowmap, title={FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent}, author={Cameron Smith and David Charatan and Ayush Tewari and Vincent Sitzmann}, year={2024}, booktitle={arXiv}, } ``` ## Acknowledgements This work was supported by the National Science Foundation under Grant No. 2211259, by the Singapore DSTA under DST00OECI20300823 (New Representations for Vision), by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) under 140D0423C0075, by the Amazon Science Hub, and by IBM. The Toyota Research Institute also partially supported this work. The views and conclusions contained herein reflect the opinions and conclusions of its authors and no other entity. ", Assign "at most 3 tags" to the expected json: {"id":"9692","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"