base on Vision agent <div align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://github.com/landing-ai/vision-agent/blob/main/assets/logo_light.svg?raw=true"> <source media="(prefers-color-scheme: light)" srcset="https://github.com/landing-ai/vision-agent/blob/main/assets/logo_dark.svg?raw=true"> <img alt="VisionAgent" height="200px" src="https://github.com/landing-ai/vision-agent/blob/main/assets/logo_light.svg?raw=true"> </picture> [](https://discord.gg/wPdN8RCYew)  [](https://badge.fury.io/py/vision-agent)  </div> ## VisionAgent VisionAgent is a library that helps you utilize agent frameworks to generate code to solve your vision task. Check out our discord for updates and roadmaps! The fastest way to test out VisionAgent is to use our web application which you can find [here](https://va.landing.ai/). ## Installation ```bash pip install vision-agent ``` ```bash export ANTHROPIC_API_KEY="your-api-key" export OPENAI_API_KEY="your-api-key" ``` > **_NOTE:_** We found using both Anthropic Claude-3.5 and OpenAI o1 to be provide the best performance for VisionAgent. If you want to use a different LLM provider or only one, see 'Using Other LLM Providers' below. You will also need to set your VisionAgent API key to be able to authenticate when using the hosted vision tools that we provide through our APIs. Currently, the APIs are free to use so you will only need to get it from [here](https://va.landing.ai/account/api-key). ```bash export VISION_AGENT_API_KEY="your-api-key" ``` ## Documentation [VisionAgent Library Docs](https://landing-ai.github.io/vision-agent/) ## Examples ### Counting cans in an image You can run VisionAgent in a local Jupyter Notebook [Counting cans in an image](https://github.com/landing-ai/vision-agent/blob/main/examples/notebooks/counting_cans.ipynb) ### Generating code You can use VisionAgent to generate code to count the number of people in an image: ```python from vision_agent.agent import VisionAgentCoderV2 from vision_agent.models import AgentMessage agent = VisionAgentCoderV2(verbose=True) code_context = agent.generate_code( [ AgentMessage( role="user", content="Count the number of people in this image", media=["people.png"] ) ] ) with open("generated_code.py", "w") as f: f.write(code_context.code + "\n" + code_context.test) ``` ### Using the tools directly VisionAgent produces code that utilizes our tools. You can also use the tools directly. For example if you wanted to detect people in an image and visualize the results: ```python import vision_agent.tools as T import matplotlib.pyplot as plt image = T.load_image("people.png") dets = T.countgd_object_detection("person", image) # visualize the countgd bounding boxes on the image viz = T.overlay_bounding_boxes(image, dets) # save the visualization to a file T.save_image(viz, "people_detected.png") # display the visualization plt.imshow(viz) plt.show() ``` You can also use the tools for running on video files: ```python import vision_agent.tools as T frames_and_ts = T.extract_frames_and_timestamps("people.mp4") # extract the frames from the frames_and_ts list frames = [f["frame"] for f in frames_and_ts] # run the countgd tracking on the frames tracks = T.countgd_sam2_video_tracking("person", frames) # visualize the countgd tracking results on the frames and save the video viz = T.overlay_segmentation_masks(frames, tracks) T.save_video(viz, "people_detected.mp4") ``` ## Using Other LLM Providers You can use other LLM providers by changing `config.py` in the `vision_agent/configs` directory. For example to change to Anthropic simply just run: ```bash cp vision_agent/configs/anthropic_config.py vision_agent/configs/config.py ``` > **_NOTE:_** VisionAgent moves fast and we are constantly updating and changing the library. If you have any questions or need help, please reach out to us on our discord channel. ", Assign "at most 3 tags" to the expected json: {"id":"10861","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"