base on Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions. It supports Python's asynchronous features, enhancing performance and enabling event capturing and simultaneous web scraping. <p align="center">
<h1>๐ Pydoll: Async Web Automation in Python!</h1>
</p>
<br>
<p align="center">
<img src="https://github.com/user-attachments/assets/c4615101-d932-4e79-8a08-f50fbc686e3b" alt="Alt text" /> <br><br>
</p>
<p align="center">
<a href="https://codecov.io/gh/autoscrape-labs/pydoll">
<img src="https://codecov.io/gh/autoscrape-labs/pydoll/graph/badge.svg?token=40I938OGM9"/>
</a>
<img src="https://github.com/thalissonvs/pydoll/actions/workflows/tests.yml/badge.svg" alt="Tests">
<img src="https://github.com/thalissonvs/pydoll/actions/workflows/ruff-ci.yml/badge.svg" alt="Ruff CI">
<img src="https://github.com/thalissonvs/pydoll/actions/workflows/release.yml/badge.svg" alt="Release">
<img src="https://tokei.rs/b1/github/thalissonvs/pydoll" alt="Total lines">
<img src="https://tokei.rs/b1/github/thalissonvs/pydoll?category=files" alt="Files">
<img src="https://tokei.rs/b1/github/thalissonvs/pydoll?category=comments" alt="Comments">
<img src="https://img.shields.io/github/issues/thalissonvs/pydoll?label=Issues" alt="GitHub issues">
<img src="https://img.shields.io/github/issues-closed/thalissonvs/pydoll?label=Closed issues" alt="GitHub closed issues">
<img src="https://img.shields.io/github/issues/thalissonvs/pydoll/bug?label=Bugs&color=red" alt="GitHub bug issues">
<img src="https://img.shields.io/github/issues/thalissonvs/pydoll/enhancement?label=Enhancements&color=purple" alt="GitHub enhancement issues">
</p>
<p align="center">
<a href="https://trendshift.io/repositories/13125" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13125" alt="thalissonvs%2Fpydoll | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<p align="center">
<b>Pydoll</b> is revolutionizing browser automation! Unlike other solutions, it <b>eliminates the need for webdrivers</b>,
providing a smooth and reliable automation experience with native asynchronous performance.
</p>
<p align="center">
<a href="#-installation">Installation</a> โข
<a href="#-quick-start">Quick Start</a> โข
<a href="#-core-components">Core Components</a> โข
<a href="#-whats-new">What's New</a> โข
<a href="#-advanced-features">Advanced Features</a>
</p>
## โจ Key Features
๐น **Zero Webdrivers!** Say goodbye to webdriver compatibility nightmares
๐น **Native Captcha Bypass!** Smoothly handles Cloudflare Turnstile and reCAPTCHA v3*
๐น **Async Performance** for lightning-fast automation
๐น **Human-like Interactions** that mimic real user behavior
๐น **Powerful Event System** for reactive automations
๐น **Multi-browser Support** including Chrome and Edge
> *Note: For Cloudflare captcha, click the checkbox by finding the div containing the iframe and using the `.click()` method. Automatic detection coming soon!
## ๐ฅ Installation
```bash
pip install pydoll-python
```
## โก Quick Start
Get started with just a few lines of code:
```python
import asyncio
from pydoll.browser.chrome import Chrome
from pydoll.constants import By
async def main():
async with Chrome() as browser:
await browser.start()
page = await browser.get_page()
# Works with captcha-protected sites
await page.go_to('https://example-with-cloudflare.com')
button = await page.find_element(By.CSS_SELECTOR, 'button')
await button.click()
asyncio.run(main())
```
Need to configure your browser? Easy!
```python
from pydoll.browser.chrome import Chrome
from pydoll.browser.options import Options
options = Options()
# Add a proxy
options.add_argument('--proxy-server=username:password@ip:port')
# Custom browser location
options.binary_location = '/path/to/your/browser'
async with Chrome(options=options) as browser:
await browser.start()
# Your code here
```
## ๐ What's New
Version 1.4.0 comes packed with amazing new features:
### ๐ค Advanced Keyboard Control
Full keyboard simulation thanks to [@cleitonleonel](https://github.com/cleitonleonel):
```python
import asyncio
from pydoll.browser.chrome import Chrome
from pydoll.browser.options import Options
from pydoll.common.keys import Keys
from pydoll.constants import By
async def main():
async with Chrome() as browser:
await browser.start()
page = await browser.get_page()
await page.go_to('https://example.com')
input_field = await page.find_element(By.CSS_SELECTOR, 'input')
await input_field.click()
# Realistic typing with customizable speed
await input_field.type_keys("
[email protected]", interval=0.2)
# Special key combinations
await input_field.key_down(Keys.SHIFT)
await input_field.send_keys("UPPERCASE")
await input_field.key_up(Keys.SHIFT)
# Navigation keys
await input_field.send_keys(Keys.ENTER)
await input_field.send_keys(Keys.PAGEDOWN)
asyncio.run(main())
```
### ๐ File Upload Support
[@yie1d](https://github.com/yie1d) brings seamless file uploads:
```python
# For input elements
file_input = await page.find_element(By.XPATH, '//input[@type="file"]')
await file_input.set_input_files('path/to/file.pdf') # Single file
await file_input.set_input_files(['file1.pdf', 'file2.jpg']) # Multiple files
# For other elements using the file chooser
async with page.expect_file_chooser(files='path/to/file.pdf'):
upload_button = await page.find_element(By.ID, 'upload-button')
await upload_button.click()
```
### ๐ Microsoft Edge Support
Now with Edge browser support thanks to [@Harris-H](https://github.com/Harris-H):
```python
import asyncio
from pydoll.browser import Edge
from pydoll.browser.options import EdgeOptions
async def main():
options = EdgeOptions()
# options.add_argument('--headless')
async with Edge(options=options) as browser:
await browser.start()
page = await browser.get_page()
await page.go_to('https://example.com')
asyncio.run(main())
```
## ๐ฏ Core Components
Pydoll offers three main interfaces for browser automation:
### Browser Interface
The Browser interface provides global control over the entire browser instance:
```python
async def browser_demo():
async with Chrome() as browser:
await browser.start()
# Create multiple pages
pages = [await browser.get_page() for _ in range(3)]
# Control the browser window
await browser.set_window_maximized()
# Manage cookies globally
await browser.set_cookies([{
'name': 'session',
'value': '12345',
'domain': 'example.com'
}])
```
#### Key Browser Methods
| Method | Description | Example |
|--------|-------------|---------|
| `async start()` | ๐ฅ Launch your browser and prepare for automation | `await browser.start()` |
| `async stop()` | ๐ Close the browser gracefully when finished | `await browser.stop()` |
| `async get_page()` | โจ Get an existing page or create a new one | `page = await browser.get_page()` |
| `async new_page(url='')` | ๐ Create a new page in the browser | `page_id = await browser.new_page()` |
| `async get_page_by_id(page_id)` | ๐ Find and control a specific page by ID | `page = await browser.get_page_by_id(id)` |
| `async get_targets()` | ๐ฏ List all open pages in the browser | `targets = await browser.get_targets()` |
| `async set_window_bounds(bounds)` | ๐ Size and position the browser window | `await browser.set_window_bounds({'width': 1024})` |
| `async set_window_maximized()` | ๐ช Maximize the browser window | `await browser.set_window_maximized()` |
| `async get_cookies()` | ๐ช Get all browser cookies | `cookies = await browser.get_cookies()` |
| `async set_cookies(cookies)` | ๐ง Set custom cookies for authentication | `await browser.set_cookies([{...}])` |
| `async delete_all_cookies()` | ๐งน Clear all cookies for a fresh state | `await browser.delete_all_cookies()` |
| `async set_download_path(path)` | ๐ Configure where downloaded files are saved | `await browser.set_download_path('/downloads')` |
### Page Interface
The Page interface lets you control individual browser tabs and interact with web content:
```python
async def page_demo():
page = await browser.get_page()
# Navigation
await page.go_to('https://example.com')
await page.refresh()
# Get page info
url = await page.current_url
html = await page.page_source
# Screenshots and PDF
await page.get_screenshot('screenshot.png')
await page.print_to_pdf('page.pdf')
# Execute JavaScript
title = await page.execute_script('return document.title')
```
#### Key Page Methods
| Method | Description | Example |
|--------|-------------|---------|
| `async go_to(url, timeout=300)` | ๐ Navigate to a URL with loading detection | `await page.go_to('https://example.com')` |
| `async refresh()` | ๐ Reload the current page | `await page.refresh()` |
| `async close()` | ๐ช Close the current tab | `await page.close()` |
| `async current_url` | ๐งญ Get the current page URL | `url = await page.current_url` |
| `async page_source` | ๐ Get the page's HTML content | `html = await page.page_source` |
| `async get_screenshot(path)` | ๐ธ Save a screenshot of the page | `await page.get_screenshot('shot.png')` |
| `async print_to_pdf(path)` | ๐ Convert the page to a PDF document | `await page.print_to_pdf('page.pdf')` |
| `async has_dialog()` | ๐ Check if a dialog is present | `if await page.has_dialog():` |
| `async accept_dialog()` | ๐ Dismiss alert and confirmation dialogs | `await page.accept_dialog()` |
| `async execute_script(script, element)` | โก Run JavaScript code on the page | `await page.execute_script('alert("Hi!")')` |
| `async get_network_logs(matches=[])` | ๐ธ๏ธ Monitor network requests | `logs = await page.get_network_logs()` |
| `async find_element(by, value)` | ๐ Find an element on the page | `el = await page.find_element(By.ID, 'btn')` |
| `async find_elements(by, value)` | ๐ Find multiple elements matching a selector | `items = await page.find_elements(By.CSS, 'li')` |
| `async wait_element(by, value, timeout=10)` | โณ Wait for an element to appear | `await page.wait_element(By.ID, 'loaded', 5)` |
### WebElement Interface
The WebElement interface provides methods to interact with DOM elements:
```python
async def element_demo():
# Find elements
button = await page.find_element(By.CSS_SELECTOR, 'button.submit')
input_field = await page.find_element(By.ID, 'username')
# Get properties
button_text = await button.get_element_text()
is_button_enabled = button.is_enabled
input_value = input_field.value
# Interact with elements
await button.scroll_into_view()
await input_field.type_keys("user123")
await button.click()
```
#### Key WebElement Methods
| Method | Description | Example |
|--------|-------------|---------|
| `value` | ๐ฌ Get the value of an input element | `value = input_field.value` |
| `class_name` | ๐จ Get the element's CSS classes | `classes = element.class_name` |
| `id` | ๐ท๏ธ Get the element's ID attribute | `id = element.id` |
| `is_enabled` | โ
Check if the element is enabled | `if button.is_enabled:` |
| `async bounds` | ๐ Get the element's position and size | `coords = await element.bounds` |
| `async inner_html` | ๐งฉ Get the element's inner HTML content | `html = await element.inner_html` |
| `async get_element_text()` | ๐ Get the element's text content | `text = await element.get_element_text()` |
| `get_attribute(name)` | ๐ Get any attribute from the element | `href = link.get_attribute('href')` |
| `async scroll_into_view()` | ๐๏ธ Scroll the element into viewport | `await element.scroll_into_view()` |
| `async click(x_offset=0, y_offset=0)` | ๐ Click the element with optional offsets | `await button.click()` |
| `async click_using_js()` | ๐ฎ Click using JavaScript for hidden elements | `await overlay_button.click_using_js()` |
| `async send_keys(text)` | โจ๏ธ Send text to input fields | `await input.send_keys("text")` |
| `async type_keys(text, interval=0.1)` | ๐จโ๐ป Type text with realistic timing | `await input.type_keys("hello", 0.2)` |
| `async get_screenshot(path)` | ๐ท Take a screenshot of the element | `await error.get_screenshot('error.png')` |
| `async set_input_files(files)` | ๐ค Upload files with file inputs | `await input.set_input_files('file.pdf')` |
## ๐ Advanced Features
### Event System
Pydoll's powerful event system lets you react to browser events in real-time:
```python
from pydoll.events.page import PageEvents
from pydoll.events.network import NetworkEvents
from functools import partial
# Page navigation events
async def on_page_loaded(event):
print(f"๐ Page loaded: {event['params'].get('url')}")
await page.enable_page_events()
await page.on(PageEvents.PAGE_LOADED, on_page_loaded)
# Network request monitoring
async def on_request(page, event):
url = event['params']['request']['url']
print(f"๐ Request to: {url}")
await page.enable_network_events()
await page.on(NetworkEvents.REQUEST_WILL_BE_SENT, partial(on_request, page))
# DOM change monitoring
from pydoll.events.dom import DomEvents
await page.enable_dom_events()
await page.on(DomEvents.DOCUMENT_UPDATED, lambda e: print("DOM updated!"))
```
### Request Interception
Pydoll gives you the power to intercept and modify network requests before they're sent! This allows you to customize headers or modify request data on the fly.
#### Basic Request Modification
The request interception system lets you monitor and modify requests before they're sent:
```python
from pydoll.events.fetch import FetchEvents
from pydoll.commands.fetch import FetchCommands
from functools import partial
async def request_interceptor(page, event):
request_id = event['params']['requestId']
url = event['params']['request']['url']
print(f"๐ Intercepted request to: {url}")
# Continue the request normally
await page._execute_command(
FetchCommands.continue_request(
request_id=request_id
)
)
# Enable interception and register your handler
await page.enable_fetch_events()
await page.on(FetchEvents.REQUEST_PAUSED, partial(request_interceptor, page))
```
#### Adding Custom Headers
Inject authentication or tracking headers into specific requests:
```python
async def auth_header_interceptor(page, event):
request_id = event['params']['requestId']
url = event['params']['request']['url']
# Only add auth headers to API requests
if '/api/' in url:
# Get the original headers
original_headers = event['params']['request'].get('headers', {})
# Add your custom headers
custom_headers = {
**original_headers,
'Authorization': 'Bearer your-token-123',
'X-Custom-Track': 'pydoll-automation'
}
await page._execute_command(
FetchCommands.continue_request(
request_id=request_id,
headers=custom_headers
)
)
else:
# Continue normally for non-API requests
await page._execute_command(
FetchCommands.continue_request(
request_id=request_id
)
)
await page.enable_fetch_events()
await page.on(FetchEvents.REQUEST_PAUSED, partial(auth_header_interceptor, page))
```
#### Modifying Request Body
Change POST data before it's sent:
```python
async def modify_request_body(page, event):
request_id = event['params']['requestId']
url = event['params']['request']['url']
method = event['params']['request'].get('method', '')
# Only modify POST requests to specific endpoints
if method == 'POST' and 'submit-form' in url:
# Get original request body if it exists
original_body = event['params']['request'].get('postData', '{}')
# In a real scenario, you'd parse and modify the body
# For this example, we're just replacing it
new_body = '{"modified": true, "data": "enhanced-by-pydoll"}'
print(f"โ๏ธ Modifying POST request to: {url}")
await page._execute_command(
FetchCommands.continue_request(
request_id=request_id,
post_data=new_body
)
)
else:
# Continue normally for other requests
await page._execute_command(
FetchCommands.continue_request(
request_id=request_id
)
)
await page.enable_fetch_events()
await page.on(FetchEvents.REQUEST_PAUSED, partial(modify_request_body, page))
```
### Filtering Request Types
You can focus on specific types of requests to intercept:
```python
# Just intercept XHR requests
await page.enable_fetch_events(resource_type='xhr')
# Or focus on document requests
await page.enable_fetch_events(resource_type='document')
# Or maybe just images
await page.enable_fetch_events(resource_type='image')
```
Available resource types include: `document`, `stylesheet`, `image`, `media`, `font`, `script`, `texttrack`, `xhr`, `fetch`, `eventsource`, `websocket`, `manifest`, `other`.
### Concurrent Automation
Process multiple pages simultaneously for maximum efficiency:
```python
async def process_page(url):
page = await browser.get_page()
await page.go_to(url)
# Do your scraping or automation here
return await page.get_element_text()
# Process multiple URLs concurrently
urls = ['https://example1.com', 'https://example2.com', 'https://example3.com']
results = await asyncio.gather(*(process_page(url) for url in urls))
```
## ๐ก Best Practices
Maximize your Pydoll experience with these tips:
โ
**Embrace async patterns** throughout your code for best performance
โ
**Use specific selectors** (IDs, unique attributes) for reliable element finding
โ
**Implement proper error handling** with try/except blocks around critical operations
โ
**Leverage the event system** instead of polling for state changes
โ
**Properly close resources** with async context managers
โ
**Wait for elements** instead of fixed sleep delays
โ
**Use realistic interactions** like `type_keys()` to avoid detection
## ๐ค Contributing
We'd love your help making Pydoll even better! Check out our [contribution guidelines](CONTRIBUTING.md) to get started. Whether it's fixing bugs, adding features, or improving documentation - all contributions are welcome!
Please make sure to:
- Write tests for new features or bug fixes
- Follow coding style and conventions
- Use conventional commits for pull requests
- Run lint and test checks before submitting
## ๐ฎ Coming Soon
Get ready for these upcoming features in Pydoll:
๐น **Auto-detection of Cloudflare Captcha** - Automatic solving without manual intervention
๐น **Fingerprint Generation & Rotation** - Dynamic browser fingerprints to avoid detection
๐น **Proxy Rotation** - Seamless IP switching for extended scraping sessions
๐น **Shadow DOM Access** - Navigate and interact with Shadow Root elements
Stay tuned and star the repository to get updates when these features are released!
## ๐ License
Pydoll is licensed under the [MIT License](LICENSE).
---
<p align="center">
<b>Pydoll</b> โ Making browser automation magical! โจ
</p>
", Assign "at most 3 tags" to the expected json: {"id":"13125","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"