base on 🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3) # LLaVA++: Extending Visual Capabilities with LLaMA-3 and Phi-3 <p align="center"> <img src="https://i.imgur.com/waxVImv.png" alt="Oryx Models"> </p> #### [Hanoona Rasheed](https://www.hanoonarasheed.com/)\*, [Muhammad Maaz](https://www.muhammadmaaz.com)\*, [Salman Khan](https://salman-h-khan.github.io/), and [Fahad Khan](https://sites.google.com/view/fahadkhans/home) \* Equal contributions #### **Mohamed bin Zayed University of AI (MBZUAI)** [](https://colab.research.google.com/drive/10Z2HaY5zvy2GZZ4v245PtiDPukm0NbF6?usp=sharing) [](https://bengal-eminent-wasp.ngrok-free.app) [](https://huggingface.co/spaces/MBZUAI/LLaMA-3-V) [](https://huggingface.co/spaces/MBZUAI/Phi-3-V) --- ## 📢 Latest Updates - **Apr-30-24**- LLaMA-3-V and Phi-3-V demos are now available via Hugging Face Spaces. Check them out at [LLaMA-3-V](https://huggingface.co/spaces/MBZUAI/LLaMA-3-V) & [Phi-3-V](https://huggingface.co/spaces/MBZUAI/Phi-3-V) 🔥🔥🔥 - **Apr-28-24**- Online demo of Phi-3-V and LLaMA-3-V are released, check them out at [Online Demo](https://bengal-eminent-wasp.ngrok-free.app) 🔥🔥🔥 - **Apr-28-24**- LoRA, fully fine-tuned and [S²](https://github.com/bfshi/scaling_on_scales.git) fine-tuned models and results are added! 🔥🔥🔥 - **Apr-27-24**- Google Colab is released to chat with Phi-3-V-3.8B model, check it out at [Google Colab](https://colab.research.google.com/drive/10Z2HaY5zvy2GZZ4v245PtiDPukm0NbF6?usp=sharing) 🔥🔥🔥 - **Apr-26-24**- Phi-3-V and LLaVA-3-V released: Excited to release the new integration of LLaVA with Phi-3 Mini Instruct and LLaMA-3 Instruct models! [Hugging Face](https://huggingface.co/collections/MBZUAI/llava-662b38b972e3e3e4d8f821bb) 🔥🔥🔥 --- <p align="center"> <img src="images/logos/face.png" width="300"> </p> ## 💬 Introduction This repository enhances the capabilities of the LLaVA 1.5 model, incorporating latest LLMs released this weak🔥, [Phi-3 Mini Instruct 3.8B](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct), and [LLaMA-3 Instruct 8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B). ## 🏆 Results: Phi-3-V and LLaVA-3-V <p align="center"> <img src="images/lava++_radar_plot.png" width="500"> </p> ### Comparison on Benchmarks for Instruction-following LMMS & academic-task-oriented datasets: <p align="center"> <img src="images/LLaVA-pp-results.png"> </p> - Average computed excluding MME, and second-best are underlined. ## 🤖 Model-Zoo The following table provides an overview of the available models in our zoo. For each model, you can find links to its Hugging Face page. | Model Name | Hugging Face Link | Summary | |---------------------------------------|:--------------------------------------------------------------------------:|-------------------------------------------------------------------------------------------------------------------| | LLaVA-Phi-3-mini-4k-instruct-pretrain | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Phi-3-mini-4k-instruct-pretrain) | Pretrained on [LCS-558K](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain). | | LLaVA-Phi-3-mini-4k-instruct-lora | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Phi-3-mini-4k-instruct-lora) | LoRA weights fine-tuned on [LLaVA-Instruct-665K](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K). | | LLaVA-Phi-3-mini-4k-instruct | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Phi-3-mini-4k-instruct) | Merged LoRA weights in HuggingFace format. | | LLaVA-Phi-3-mini-4k-instruct-FT | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Phi-3-mini-4k-instruct-FT) | Fully fine-tuned model weights in HuggingFace format. | | Model Name | Hugging Face Link | Summary | |-----------------------------------------|:-------------------------------------------------------------------------------------:|-------------------------------------------------------------------------------------------------------------------| | LLaVA-Meta-Llama-3-8B-Instruct-pretrain | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Meta-Llama-3-8B-Instruct-pretrain) | Pretrained on [LCS-558K](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain). | | LLaVA-Meta-Llama-3-8B-Instruct-lora | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Meta-Llama-3-8B-Instruct-lora) | LoRA weights fine-tuned on [LLaVA-Instruct-665K](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K). | | LLaVA-Meta-Llama-3-8B-Instruct | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Meta-Llama-3-8B-Instruct) | Merged weights in HuggingFace format. | | LLaVA-Meta-Llama-3-8B-Instruct-FT | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Meta-Llama-3-8B-Instruct-FT) | Fully fine-tuned model weights in HuggingFace format. | | LLaVA-Meta-Llama-3-8B-Instruct-FT-S2 | [Hugging Face](https://huggingface.co/MBZUAI/LLaVA-Meta-Llama-3-8B-Instruct-FT-S2) | Fully fine-tuned S2 model weights in HuggingFace format. | # Installation ```bash git clone https://github.com/mbzuai-oryx/LLaVA-pp.git cd LLaVA-pp git submodule update --init --recursive ``` Packages you need to update from LLAVA: ```bash pip install git+https://github.com/huggingface/transformers@a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3 ``` ## 🚀 Phi-3-V To integrate Phi-3-V with LLaVA, follow these steps to update the codebase: ```bash # Copy necessary files cp Phi-3-V/train.py LLaVA/llava/train/train.py cp Phi-3-V/llava_phi3.py LLaVA/llava/model/language_model/llava_phi3.py cp Phi-3-V/builder.py LLaVA/llava/model/builder.py cp Phi-3-V/model__init__.py LLaVA/llava/model/__init__.py cp Phi-3-V/main__init__.py LLaVA/llava/__init__.py cp Phi-3-V/conversation.py LLaVA/llava/conversation.py # Training commands cp scripts/Phi3-V_pretrain.sh LLaVA/Vi-phi3_pretrain.sh cp scripts/Phi3-V_finetune_lora.sh LLaVA/Vi-phi3_finetune_lora.sh ``` ### Train Phi-3-V 1. Pre-train ```bash cd LLaVA bash Phi3-V_pretrain.sh ``` 2. Finetune ```bash cd LLaVA bash Phi3-V_finetune_lora.sh ``` ## 🚀 LLaMA-3-V To integrate LLaMA-3-V with LLaVA, follow these steps to update the codebase: ```bash # Copy necessary files cp LLaMA-3-V/train.py LLaVA/llava/train/train.py cp LLaMA-3-V/conversation.py LLaVA/llava/conversation.py cp LLaMA-3-V/builder.py LLaVA/llava/model/builder.py cp LLaMA-3-V/llava_llama.py LLaVA/llava/model/language_model/llava_llama.py # Training commands cp scripts/LLaMA3-V_pretrain.sh LLaVA/LLaMA3-V_pretrain.sh cp scripts/LLaMA3-V_finetune_lora.sh LLaVA/LLaMA3-V_finetune_lora.sh ``` ### Train LLaMA-3-V 1. Pre-train ```bash cd LLaVA bash LLaMA3-V_pretrain.sh ``` 2. Finetune ```bash cd LLaVA bash LLaMA3-V_finetune_lora.sh ``` --- ## 🙏 Acknowledgement We are thankful to [LLaVA](https://github.com/haotian-liu/LLaVA.git), [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval.git) and [S²-Wrapper](https://github.com/bfshi/scaling_on_scales.git) for releasing their models and code as open-source contributions. In case if you face any issues or have any questions, please feel free to create an issue or reach out at [[email protected]]([email protected]) & [[email protected]]([email protected]). ## 📜 Citation ```bibtex @misc{hanoona2024LLaVA++, title={LLaVA++: Extending Visual Capabilities with LLaMA-3 and Phi-3}, author={Rasheed, Hanoona and Maaz, Muhammad and Khan, Salman and Khan, Fahad S.}, url={https://github.com/mbzuai-oryx/LLaVA-pp}, year={2024} } ``` --- [<img src="images/logos/IVAL_logo.png" width="200" height="100">](https://www.ival-mbzuai.com) [<img src="images/logos/Oryx_logo.png" width="100" height="100">](https://github.com/mbzuai-oryx) [<img src="images/logos/MBZUAI_logo.png" width="360" height="85">](https://mbzuai.ac.ae) ", Assign "at most 3 tags" to the expected json: {"id":"9712","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"