base on Code release for CVPR'24 submission 'OmniGlue' <div align="center"> # \[CVPR'24\] Code release for OmniGlue <p align="center"> <a href="https://hwjiang1510.github.io/">Hanwen Jiang</a>, <a href="https://scholar.google.com/citations?user=jgSItF4AAAAJ">Arjun Karpur</a>, <a href="https://scholar.google.com/citations?user=7EeSOcgAAAAJ">Bingyi Cao</a>, <a href="https://www.cs.utexas.edu/~huangqx/">Qixing Huang</a>, <a href="https://andrefaraujo.github.io/">Andre Araujo</a> </p> </div> -------------------------------------------------------------------------------- <div align="center"> <a href="https://hwjiang1510.github.io/OmniGlue/"><strong>Project Page</strong></a> | <a href="https://arxiv.org/abs/2405.12979"><strong>Paper</strong></a> | <a href="#installation"><strong>Usage</strong></a> | <a href="https://huggingface.co/spaces/qubvel-hf/omniglue"><strong>Demo</strong></a> </div> <br> <div align="center"> [](https://huggingface.co/spaces/qubvel-hf/omniglue) </div> <br> Official code release for the CVPR 2024 paper: **OmniGlue: Generalizable Feature Matching with Foundation Model Guidance**.  **Abstract:** The image matching field has been witnessing a continuous emergence of novel learnable feature matching techniques, with ever-improving performance on conventional benchmarks. However, our investigation shows that despite these gains, their potential for real-world applications is restricted by their limited generalization capabilities to novel image domains. In this paper, we introduce OmniGlue, the first learnable image matcher that is designed with generalization as a core principle. OmniGlue leverages broad knowledge from a vision foundation model to guide the feature matching process, boosting generalization to domains not seen at training time. Additionally, we propose a novel keypoint position-guided attention mechanism which disentangles spatial and appearance information, leading to enhanced matching descriptors. We perform comprehensive experiments on a suite of 6 datasets with varied image domains, including scene-level, object-centric and aerial images. OmniGlue’s novel components lead to relative gains on unseen domains of 18.8% with respect to a directly comparable reference model, while also outperforming the recent LightGlue method by 10.1% relatively. ## Installation First, use pip to install `omniglue`: ```sh conda create -n omniglue pip conda activate omniglue git clone https://github.com/google-research/omniglue.git cd omniglue pip install -e . ``` Then, download the following models to `./models/` ```sh # Download to ./models/ dir. mkdir models cd models # SuperPoint. git clone https://github.com/rpautrat/SuperPoint.git mv SuperPoint/pretrained_models/sp_v6.tgz . && rm -rf SuperPoint tar zxvf sp_v6.tgz && rm sp_v6.tgz # DINOv2 - vit-b14. wget https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_pretrain.pth # OmniGlue. wget https://storage.googleapis.com/omniglue/og_export.zip unzip og_export.zip && rm og_export.zip ``` Direct download links: - [[SuperPoint weights]](https://github.com/rpautrat/SuperPoint/tree/master/pretrained_models): from [github.com/rpautrat/SuperPoint](https://github.com/rpautrat/SuperPoint) - [[DINOv2 weights]](https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_pretrain.pth): from [github.com/facebookresearch/dinov2](https://github.com/facebookresearch/dinov2) (ViT-B/14 distilled backbone without register). - [[OmniGlue weights]](https://storage.googleapis.com/omniglue/og_export.zip) ## Usage The code snippet below outlines how you can perform OmniGlue inference in your own python codebase. ```py import omniglue image0 = ... # load images from file into np.array image1 = ... og = omniglue.OmniGlue( og_export='./models/og_export', sp_export='./models/sp_v6', dino_export='./models/dinov2_vitb14_pretrain.pth', ) match_kp0s, match_kp1s, match_confidences = og.FindMatches(image0, image1) # Output: # match_kp0: (N, 2) array of (x,y) coordinates in image0. # match_kp1: (N, 2) array of (x,y) coordinates in image1. # match_confidences: N-dim array of each of the N match confidence scores. ``` ## Demo `demo.py` contains example usage of the `omniglue` module. To try with your own images, replace `./res/demo1.jpg` and `./res/demo2.jpg` with your own filepaths. ```sh conda activate omniglue python demo.py ./res/demo1.jpg ./res/demo2.jpg # <see output in './demo_output.png'> ``` Expected output:  ## Repo TODOs - ~~Provide `demo.py` example usage script.~~ - ~~Add to image matching webui~~ (credit: [@Vincentqyw](https://github.com/Vincentqyw)) - Support matching for pre-extracted features. - Release eval pipelines for in-domain (MegaDepth). - Release eval pipelines for all out-of-domain datasets. ## BibTex ``` @inproceedings{jiang2024Omniglue, title={OmniGlue: Generalizable Feature Matching with Foundation Model Guidance}, author={Jiang, Hanwen and Karpur, Arjun and Cao, Bingyi and Huang, Qixing and Araujo, Andre}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2024}, } ``` -------------------------------------------------------------------------------- This is not an officially supported Google product. ", Assign "at most 3 tags" to the expected json: {"id":"10385","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"