base on streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL <div align="center"> <h1>maestro</h1> <div> <img src="https://github.com/user-attachments/assets/c9416f1f-a2bf-4590-86da-d2fc89ba559b" width="80" height="40" /> <img src="https://github.com/user-attachments/assets/75dc7214-e82a-498d-950e-c64d90218e49" width="80" height="40" /> <img src="https://github.com/user-attachments/assets/5d265473-b938-4501-b894-6a44a6a28a8c" width="80" height="40" /> <img src="https://github.com/user-attachments/assets/b7ccdf39-ac77-4dbd-8608-0fa2d9dadf0a" width="80" height="40" /> </div> <br> [](https://badge.fury.io/py/maestro) [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_qwen2_5_vl_json_extraction.ipynb) [](https://discord.gg/GbfgXGJ8Bk) </div> ## Hello **maestro** is a streamlined tool to accelerate the fine-tuning of multimodal models. By encapsulating best practices from our core modules, maestro handles configuration, data loading, reproducibility, and training loop setup. It currently offers ready-to-use recipes for popular vision-language models such as **Florence-2**, **PaliGemma 2**, and **Qwen2.5-VL**. ## Fine-tune VLMs for free | model, task and acceleration | open in colab | |:------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | Florence-2 (0.9B) object detection with LoRA (experimental) | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_florence_2_object_detection.ipynb) | | PaliGemma 2 (3B) JSON data extraction with LoRA | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_paligemma_2_json_extraction.ipynb) | | Qwen2.5-VL (3B) JSON data extraction with QLoRA | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_qwen2_5_vl_json_extraction.ipynb) | | Qwen2.5-VL (7B) object detection with QLoRA (experimental) | [](https://colab.research.google.com/github/roboflow/maestro/blob/develop/cookbooks/maestro_qwen2_5_vl_object_detection.ipynb) | ## News - `2025/02/05` (`1.0.0`): This release introduces support for Florence-2, PaliGemma 2, and Qwen2.5-VL and includes LoRA, QLoRA, and graph freezing to keep hardware requirements in check. It offers a single CLI/SDK to reduce code complexity, and a consistent JSONL format to streamline data handling. ## Quickstart ### Install To begin, install the model-specific dependencies. Since some models may have clashing requirements, we recommend creating a dedicated Python environment for each model. ```bash pip install "maestro[paligemma_2]" ``` ### CLI Kick off fine-tuning with our command-line interface, which leverages the configuration and training routines defined in each model’s core module. Simply specify key parameters such as the dataset location, number of epochs, batch size, optimization strategy, and metrics. ```bash maestro paligemma_2 train \ --dataset "dataset/location" \ --epochs 10 \ --batch-size 4 \ --optimization_strategy "qlora" \ --metrics "edit_distance" ``` ### Python For greater control, use the Python API to fine-tune your models. Import the train function from the corresponding module and define your configuration in a dictionary. The core modules take care of reproducibility, data preparation, and training setup. ```python from maestro.trainer.models.paligemma_2.core import train config = { "dataset": "dataset/location", "epochs": 10, "batch_size": 4, "optimization_strategy": "qlora", "metrics": ["edit_distance"] } train(config) ``` ## Contribution We appreciate your input as we continue refining Maestro. Your feedback is invaluable in guiding our improvements. To learn how you can help, please check out our [Contributing Guide](https://github.com/roboflow/maestro/blob/develop/CONTRIBUTING.md). If you have any questions or ideas, feel free to start a conversation in our [GitHub Discussions](https://github.com/roboflow/maestro/discussions). Thank you for being a part of our journey! ", Assign "at most 3 tags" to the expected json: {"id":"5423","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"