base on # Computer Use Preview ## Quick Start This section will guide you through setting up and running the Computer Use Preview model, either the Gemini Developer API or Vertex AI. Follow these steps to get started. ### 1. Installation **Clone the Repository** ```bash git clone https://github.com/google/computer-use-preview.git cd computer-use-preview ``` **Set up Python Virtual Environment and Install Dependencies** ```bash python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt ``` **Install Playwright and Browser Dependencies** ```bash # Install system dependencies required by Playwright for Chrome playwright install-deps chrome # Install the Chrome browser for Playwright playwright install chrome ``` ### 2. Configuration You can get started using either the Gemini Developer API or Vertex AI. #### A. If using the Gemini Developer API: You need a Gemini API key to use the agent: ```bash export GEMINI_API_KEY="YOUR_GEMINI_API_KEY" ``` Or to add this to your virtual environment: ```bash echo 'export GEMINI_API_KEY="YOUR_GEMINI_API_KEY"' >> .venv/bin/activate # After editing, you'll need to deactivate and reactivate your virtual # environment if it's already active: deactivate source .venv/bin/activate ``` Replace `YOUR_GEMINI_API_KEY` with your actual key. #### B. If using the Vertex AI Client: You need to explicitly use Vertex AI, then provide project and location to use the agent: ```bash export USE_VERTEXAI=true export VERTEXAI_PROJECT="YOUR_PROJECT_ID" export VERTEXAI_LOCATION="YOUR_LOCATION" ``` Or to add this to your virtual environment: ```bash echo 'export USE_VERTEXAI=true' >> .venv/bin/activate echo 'export VERTEXAI_PROJECT="your-project-id"' >> .venv/bin/activate echo 'export VERTEXAI_LOCATION="your-location"' >> .venv/bin/activate # After editing, you'll need to deactivate and reactivate your virtual # environment if it's already active: deactivate source .venv/bin/activate ``` Replace `YOUR_PROJECT_ID` and `YOUR_LOCATION` with your actual project and location. ### 3. Running the Tool The primary way to use the tool is via the `main.py` script. **General Command Structure:** ```bash python main.py --query "Go to Google and type 'Hello World' into the search bar" ``` **Available Environments:** You can specify a particular environment with the ```--env <environment>``` flag. Available options: - `playwright`: Runs the browser locally using Playwright. - `browserbase`: Connects to a Browserbase instance. **Local Playwright** Runs the agent using a Chrome browser instance controlled locally by Playwright. ```bash python main.py --query="Go to Google and type 'Hello World' into the search bar" --env="playwright" ``` You can also specify an initial URL for the Playwright environment: ```bash python main.py --query="Go to Google and type 'Hello World' into the search bar" --env="playwright" --initial_url="https://www.google.com/search?q=latest+AI+news" ``` **Browserbase** Runs the agent using Browserbase as the browser backend. Ensure the proper Browserbase environment variables are set:`BROWSERBASE_API_KEY` and `BROWSERBASE_PROJECT_ID`. ```bash python main.py --query="Go to Google and type 'Hello World' into the search bar" --env="browserbase" ``` ## Agent CLI The `main.py` script is the command-line interface (CLI) for running the browser agent. ### Command-Line Arguments | Argument | Description | Required | Default | Supported Environment(s) | |-|-|-|-|-| | `--query` | The natural language query for the browser agent to execute. | Yes | N/A | All | | `--env` | The computer use environment to use. Must be one of the following: `playwright`, or `browserbase` | No | N/A | All | | `--initial_url` | The initial URL to load when the browser starts. | No | https://www.google.com | All | | `--highlight_mouse` | If specified, the agent will attempt to highlight the mouse cursor's position in the screenshots. This is useful for visual debugging. | No | False (not highlighted) | `playwright` | ### Environment Variables | Variable | Description | Required | |-|-|-| | GEMINI_API_KEY | Your API key for the Gemini model. | Yes | | BROWSERBASE_API_KEY | Your API key for Browserbase. | Yes (when using the browserbase environment) | | BROWSERBASE_PROJECT_ID | Your Project ID for Browserbase. | Yes (when using the browserbase environment) | ## Known Issues ### Playwright Dropdown Menu On certain operating systems, the Playwright browser is unable to capture `<select>` elements because they are rendered by the operating system. As a result, the agent is unable to send the correct screenshot to the model. There are several ways to mitigate this. 1. Use the Browserbase option instead of Playwright. 2. Inject a script like [proxy-select](https://github.com/amitamb/proxy-select) to render a custom `<select>` element. You must inject `proxy-select.css` and `proxy-select.js` into each page that has a non-custom `<select>` element. You can do this in the [`Playwright.__enter__`](https://github.com/google-gemini/computer-use-preview/blob/main/computers/playwright/playwright.py#L100) method by adding a few lines of code, like the following (replacing `PROXY_SELECT_JS` and `PROXY_SELECT_CSS` with the appropriate variables): ```python self._page.add_init_script(PROXY_SELECT_JS) def inject_style(page): try: page.add_style_tag(content=PROXY_SELECT_CSS) except Exception as e: print(f"Error injecting style: {e}") self._page.on('domcontentloaded', inject_style) ``` Note, option 2 does not work 100% of the time, but is a temporary workaround for certain websites. The better option is to use Browserbase. ", Assign "at most 3 tags" to the expected json: {"id":"16606","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"