base on Implementation of XFeat (CVPR 2024). Do you need robust and fast local feature extraction? You are in the right place! ## XFeat: Accelerated Features for Lightweight Image Matching [Guilherme Potje](https://guipotje.github.io/) · [Felipe Cadar](https://eucadar.com/) · [Andre Araujo](https://andrefaraujo.github.io/) · [Renato Martins](https://renatojmsdh.github.io/) · [Erickson R. Nascimento](https://homepages.dcc.ufmg.br/~erickson/) [](LICENSE) [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/xfeat_matching.ipynb) [](https://huggingface.co/spaces/qubvel-hf/xfeat) ### [[ArXiv]](https://arxiv.org/abs/2404.19174) | [[Project Page]](https://www.verlab.dcc.ufmg.br/descriptors/xfeat_cvpr24/) | [[CVPR'24 Paper]](https://openaccess.thecvf.com/content/CVPR2024/html/Potje_XFeat_Accelerated_Features_for_Lightweight_Image_Matching_CVPR_2024_paper.html) - Training code is now available -> [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/XFeat_training_example.ipynb) - 🎉 **New!** XFeat + LighterGlue (smaller version of LightGlue) available! 🚀 [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/xfeat%2Blg_torch_hub.ipynb) <div align="center" style="display: flex; justify-content: center; align-items: center; flex-direction: column;"> <div style="display: flex; justify-content: space-around; width: 100%;"> <img src='./figs/xfeat.gif' width="400"/> <img src='./figs/sift.gif' width="400"/> </div> Real-time XFeat demonstration (left) compared to SIFT (right) on a textureless scene. SIFT cannot handle fast camera movements, while XFeat provides robust matches under adverse conditions, while being faster than SIFT on CPU. </div> **TL;DR**: Really fast learned keypoint detector and descriptor. Supports sparse and semi-dense matching. Just wanna quickly try on your images? Check this out: [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/xfeat_torch_hub.ipynb) [](https://huggingface.co/spaces/qubvel-hf/xfeat) ## Table of Contents - [Introduction](#introduction) <img align="right" src='./figs/xfeat_quali.jpg' width=360 /> - [Installation](#installation) - [Usage](#usage) - [Inference](#inference) - [Training](#training) - [Evaluation](#evaluation) - [Real-time demo app](#real-time-demo) - [XFeat+LightGlue](#xfeat-with-lightglue) - [Contribute](#contributing) - [Citation](#citation) - [License](#license) - [Acknowledgements](#acknowledgements) ## Introduction This repository contains the official implementation of the paper: *[XFeat: Accelerated Features for Lightweight Image Matching](https://arxiv.org/abs/2404.19174)*, to be presented at CVPR 2024. **Motivation.** Why another keypoint detector and descriptor among dozens of existing ones? We noticed that the current trend in the literature focuses on accuracy but often neglects compute efficiency, especially when deploying these solutions in the real-world. For applications in mobile robotics and augmented reality, it is critical that models can run on hardware-constrained computers. To this end, XFeat was designed as an agnostic solution focusing on both accuracy and efficiency in an image matching pipeline. **Capabilities.** - Real-time sparse inference on CPU for VGA images (tested on laptop with an i5 CPU and vanilla pytorch); - Simple architecture components which facilitates deployment on embedded devices (jetson, raspberry pi, custom AI chips, etc..); - Supports both sparse and semi-dense matching of local features; - Compact descriptors (64D); - Performance comparable to known deep local features such as SuperPoint while being significantly faster and more lightweight. Also, XFeat exhibits much better robustness to viewpoint and illumination changes than classic local features as ORB and SIFT; - Supports batched inference if you want ridiculously fast feature extraction. On VGA sparse setting, we achieved about 1,400 FPS using an RTX 4090. - For single batch inference on GPU (VGA), one can easily achieve over 150 FPS while leaving lots of room on the GPU for other concurrent tasks. ## **Paper Abstract.** We introduce a lightweight and accurate architecture for resource-efficient visual correspondence. Our method, dubbed XFeat (Accelerated Features), revisits fundamental design choices in convolutional neural networks for detecting, extracting, and matching local features. Our new model satisfies a critical need for fast and robust algorithms suitable to resource-limited devices. In particular, accurate image matching requires sufficiently large image resolutions -- for this reason, we keep the resolution as large as possible while limiting the number of channels in the network. Besides, our model is designed to offer the choice of matching at the sparse or semi-dense levels, each of which may be more suitable for different downstream applications, such as visual navigation and augmented reality. Our model is the first to offer semi-dense matching efficiently, leveraging a novel match refinement module that relies on coarse local descriptors. XFeat is versatile and hardware-independent, surpassing current deep learning-based local features in speed (up to 5x faster) with comparable or better accuracy, proven in pose estimation and visual localization. We showcase it running in real-time on an inexpensive laptop CPU without specialized hardware optimizations. **Overview of XFeat's achitecture.** XFeat extracts a keypoint heatmap $\mathbf{K}$, a compact 64-D dense descriptor map $\mathbf{F}$, and a reliability heatmap $\mathbf{R}$. It achieves unparalleled speed via early downsampling and shallow convolutions, followed by deeper convolutions in later encoders for robustness. Contrary to typical methods, it separates keypoint detection into a distinct branch, using $1 \times 1$ convolutions on an $8 \times 8$ tensor-block-transformed image for fast processing, being one of the few current learned methods that decouples detection & description and can be processed independently. <img align="center" src="./figs/xfeat_arq.png" width=1000 /> ## Timing Analyses on CPU. We show that both detection branch & match refinement module costs are small and bring significant advantages in accuracy (please check the ablation section in the paper). <img align="center" src="./figs/timings.png" width=840 /> Furthermore, XFeat performs effectively in both indoor and outdoor scenes, achieving an excellent compute-accuracy trade-off as demonstrated below. Note that in the paper, the teaser figure has a VGA resolution on the x-axis and 1,200 pixels on the y-axis. Below, we present an updated figure for improved clarity, maintaining the same x-y axis resolution. <img align="center" src="./figs/speed_accuracy.png" width=840 /> ## Installation XFeat has minimal dependencies, only relying on torch. Also, XFeat does not need a GPU for real-time sparse inference (vanilla pytorch w/o any special optimization), unless you run it on high-res images. If you want to run the real-time matching demo, you will also need OpenCV. We recommend using conda, but you can use any virtualenv of your choice. If you use conda, just create a new env with: ```bash git clone https://github.com/verlab/accelerated_features.git cd accelerated_features #Create conda env conda create -n xfeat python=3.8 conda activate xfeat ``` Then, install [pytorch (>=1.10)](https://pytorch.org/get-started/previous-versions/) and then the rest of depencencies in case you want to run the demos: ```bash #CPU only, for GPU check in pytorch website the most suitable version to your gpu. pip install torch==1.10.1+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html # CPU only for MacOS # pip install torch==1.10.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html #Install dependencies for the demo pip install opencv-contrib-python tqdm ``` ## Usage For your convenience, we provide ready to use notebooks for some examples. | **Description** | **Notebook** | |--------------------------------|-------------------------------| | Minimal example | [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/minimal_example.ipynb) | | Matching & registration example | [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/xfeat_matching.ipynb) | | Torch hub example | [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/xfeat_torch_hub.ipynb) | | Training example (synthetic) | [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/XFeat_training_example.ipynb) | | XFeat + LightGlue | [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/xfeat%2Blg_torch_hub.ipynb) | ### Inference To run XFeat on an image, three lines of code is enough: ```python from modules.xfeat import XFeat xfeat = XFeat() #Simple inference with batch sz = 1 output = xfeat.detectAndCompute(torch.randn(1,3,480,640), top_k = 4096)[0] ``` Or you can use this [script](./minimal_example.py) in the root folder: ```bash python3 minimal_example.py ``` If you already have pytorch, simply use torch hub if you like it: ```python import torch xfeat = torch.hub.load('verlab/accelerated_features', 'XFeat', pretrained = True, top_k = 4096) #Simple inference with batch sz = 1 output = xfeat.detectAndCompute(torch.randn(1,3,480,640), top_k = 4096)[0] ``` ### Training [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/XFeat_training_example.ipynb) To train XFeat as described in the paper, you will need MegaDepth & COCO_20k subset of COCO2017 dataset. You can obtain the full COCO2017 train data at https://cocodataset.org/. However, we [make available](https://drive.google.com/file/d/1ijYsPq7dtLQSl-oEsUOGH1fAy21YLc7H/view?usp=drive_link) a subset of COCO for convenience. We simply selected a subset of 20k images according to image resolution. Please check COCO [terms of use](https://cocodataset.org/#termsofuse) before using the data. To reproduce the training setup from the paper, please follow the steps: 1. Download [COCO_20k](https://drive.google.com/file/d/1ijYsPq7dtLQSl-oEsUOGH1fAy21YLc7H/view?usp=drive_link) containing a subset of COCO2017; 2. Download MegaDepth dataset. You can follow [LoFTR instructions](https://github.com/zju3dv/LoFTR/blob/master/docs/TRAINING.md#download-datasets), we use the same standard as LoFTR. Then put the megadepth indices inside the MegaDepth root folder following the standard below: ```bash {megadepth_root_path}/train_data/megadepth_indices #indices {megadepth_root_path}/MegaDepth_v1 #images & depth maps & poses ``` 3. Finally you can call training ```bash python3 -m modules.training.train --training_type xfeat_default --megadepth_root_path <path_to>/MegaDepth --synthetic_root_path <path_to>/coco_20k --ckpt_save_path /path/to/ckpts ``` ### Evaluation ---- **MegaDepth-1500** Please note that due to the stochastic nature of RANSAC and major code refactoring, you may observe slightly different AuC results; however, they should be very close to those reported in the paper. To evaluate on the MegaDepth dataset, you need to first get the dataset: ```bash python3 -m modules.dataset.download --megadepth-1500 --download_dir </path/to/desired/folder> ``` Then, you call the mega1500 eval script, you can choose between `xfeat, xfeat-star and alike`. It should take about a minute to run the benchmark: ```bash python3 -m modules.eval.megadepth1500 --dataset-dir </data/Mega1500> --matcher xfeat --ransac-thr 2.5 ``` --- **ScanNet-1500** To evaluate on the ScanNet eval dataset, you need to first get the dataset: ```bash python3 -m modules.dataset.download --scannet-1500 --download_dir </path/to/desired/folder> ``` Then, you can call the scannet1500 eval script, it should take a couple of minutes: ```bash python3 -m modules.eval.scannet1500 --scannet_path </data/ScanNet1500> --output </data/ScanNet1500/output> && python3 -m modules.eval.scannet1500 --scannet_path </data/ScanNet1500> --output </data/ScanNet1500/output> --show ``` --- ## Real-time Demo To demonstrate the capabilities of XFeat, we provide a real-time matching demo with Homography registration. Currently, you can experiment with XFeat, ORB and SIFT. You will need a working webcam. To run the demo and show the possible input flags, please run: ```bash python3 realtime_demo.py -h ``` Don't forget to press 's' to set a desired reference image. Notice that the demo only works correctly for planar scenes and rotation-only motion, because we're using a homography model. If you want to run the demo with XFeat, please run: ```bash python3 realtime_demo.py --method XFeat ``` Or test with SIFT or ORB: ```bash python3 realtime_demo.py --method SIFT python3 realtime_demo.py --method ORB ``` ## XFeat with LightGlue We have trained a lighter version of LightGlue (LighterGlue). It has fewer parameters and is approximately three times faster than the original LightGlue. Special thanks to the developers of the [GlueFactory](https://github.com/cvg/glue-factory) library, which enabled us to train this version of LightGlue with XFeat. Below, we compare the original SP + LG using the [GlueFactory](https://github.com/cvg/glue-factory) evaluation script on MegaDepth-1500. Please follow the example to test on your own images: [](https://colab.research.google.com/github/verlab/accelerated_features/blob/main/notebooks/xfeat%2Blg_torch_hub.ipynb) Metrics (AUC @ 5 / 10 / 20) | Setup | Max Dimension | Keypoints | XFeat + LighterGlue | SuperPoint + LightGlue (Official) | |-----------------|---------------|-----------|-------------------------------|-----------------------------------| | **Fast** | 640 | 1300 | 0.444 / 0.610 / 0.746 | 0.469 / 0.633 / 0.762 | | **Accurate** | 1024 | 4096 | 0.564 / 0.710 / 0.819 | 0.591 / 0.738 / 0.841 | ## Contributing Contributions to XFeat are welcome! Currently, it would be nice to have an export script to efficient deployment engines such as TensorRT and ONNX. Also, it would be cool to train other lightweight learned matchers on top of XFeat local features. ## Citation If you find this code useful for your research, please cite the paper: ```bibtex @INPROCEEDINGS{potje2024cvpr, author={Guilherme {Potje} and Felipe {Cadar} and Andre {Araujo} and Renato {Martins} and Erickson R. {Nascimento}}, booktitle={2024 IEEE / CVF Computer Vision and Pattern Recognition (CVPR)}, title={XFeat: Accelerated Features for Lightweight Image Matching}, year={2024}} ``` ## License [](LICENSE) ## Acknowledgements - We thank the agencies CAPES, CNPq, and Google for funding different parts of this work. - We thank the developers of Kornia for the [kornia library](https://github.com/kornia/kornia)! **VeRLab:** Laboratory of Computer Vison and Robotics https://www.verlab.dcc.ufmg.br <br> <img align="left" width="auto" height="50" src="./figs/ufmg.png"> <img align="right" width="auto" height="50" src="./figs/verlab.png"> <br/> ", Assign "at most 3 tags" to the expected json: {"id":"9883","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"