base on Official code for "FeatUp: A Model-Agnostic Frameworkfor Features at Any Resolution" ICLR 2024 # FeatUp: A Model-Agnostic Framework for Features at Any Resolution ### ICLR 2024 [](https://aka.ms/featup) [](https://arxiv.org/abs/2403.10516) [](https://colab.research.google.com/github/mhamilton723/FeatUp/blob/main/example_usage.ipynb) [](https://huggingface.co/spaces/mhamilton723/FeatUp) [](https://huggingface.co/papers/2403.10516) [](https://paperswithcode.com/sota/feature-upsampling-on-imagenet?p=featup-a-model-agnostic-framework-for) [Stephanie Fu*](https://stephanie-fu.github.io/), [Mark Hamilton*](https://mhamilton.net/), [Laura Brandt](https://people.csail.mit.edu/lebrandt/), [Axel Feldman](https://feldmann.nyc/), [Zhoutong Zhang](https://ztzhang.info/), [William T. Freeman](https://billf.mit.edu/about/bio) *Equal Contribution.  *TL;DR*:FeatUp improves the spatial resolution of any model's features by 16-32x without changing their semantics. https://github.com/mhamilton723/FeatUp/assets/6456637/8fb5aa7f-4514-4a97-aebf-76065163cdfd ## Contents <!--ts--> * [Install](#install) * [Using Pretrained Upsamplers](#using-pretrained-upsamplers) * [Fitting an Implicit Upsampler](#fitting-an-implicit-upsampler-to-an-image) * [Coming Soon](coming-soon) * [Citation](#citation) * [Contact](#contact) <!--te--> ## Install ### Pip For those just looking to quickly use the FeatUp APIs install via: ```shell script pip install git+https://github.com/mhamilton723/FeatUp ``` ### Local Development To install FeatUp for local development and to get access to the sample images install using the following: ```shell script git clone https://github.com/mhamilton723/FeatUp.git cd FeatUp pip install -e . ``` ## Using Pretrained Upsamplers To see examples of pretrained model usage please see our [Collab notebook](https://colab.research.google.com/github/mhamilton723/FeatUp/blob/main/example_usage.ipynb). We currently supply the following pretrained versions of FeatUp's JBU upsampler: | Model Name | Checkpoint | Checkpoint (No LayerNorm) | Torch Hub Repository | Torch Hub Name | |------------|----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|----------------------|----------------| | DINO | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/dino16_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/dino16_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp | dino16 | | DINO v2 | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/dinov2_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/dinov2_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp | dinov2 | | CLIP | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/clip_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/clip_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp | clip | | MaskCLIP | n/a | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/maskclip_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp | maskclip | | ViT | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/vit_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/vit_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp | vit | | ResNet50 | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/resnet50_jbu_stack_cocostuff.ckpt) | [Download](https://marhamilresearch4.blob.core.windows.net/feature-upsampling-public/pretrained/no_norm/resnet50_jbu_stack_cocostuff.ckpt) | mhamilton723/FeatUp | resnet50 | For example, to load the FeatUp JBU upsampler for the DINO backbone without an additional LayerNorm on the spatial features: ```python upsampler = torch.hub.load("mhamilton723/FeatUp", 'dino16', use_norm=False) ``` To load upsamplers trained on backbones with additional LayerNorm operations which makes training and transfer learning a bit more stable: ```python upsampler = torch.hub.load("mhamilton723/FeatUp", 'dino16') ``` ## Fitting an Implicit Upsampler to an Image To train an implicit upsampler for a given image and backbone first clone the repository and install it for [local development](#local-development). Then run ```python cd featup python train_implicit_upsampler.py ``` Parameters for this training operation can be found in the [implicit_upsampler config file](featup/configs/implicit_upsampler.yaml). ## Local Gradio Demo To run our [HuggingFace Spaces hosted FeatUp demo](https://huggingface.co/spaces/mhamilton723/FeatUp) locally first install FeatUp for local development. Then run: ```shell python gradio_app.py ``` Wait a few seconds for the demo to spin up, then navigate to [http://localhost:7860/](http://localhost:7860/) to view the demo. ## Coming Soon: - Training your own FeatUp joint bilateral upsampler - Simple API for Implicit FeatUp training ## Citation ``` @inproceedings{ fu2024featup, title={FeatUp: A Model-Agnostic Framework for Features at Any Resolution}, author={Stephanie Fu and Mark Hamilton and Laura E. Brandt and Axel Feldmann and Zhoutong Zhang and William T. Freeman}, booktitle={The Twelfth International Conference on Learning Representations}, year={2024}, url={https://openreview.net/forum?id=GkJiNn2QDF} } ``` ## Contact For feedback, questions, or press inquiries please contact [Stephanie Fu](mailto:[email protected]) and [Mark Hamilton](mailto:[email protected]) ", Assign "at most 3 tags" to the expected json: {"id":"8737","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"