taskPyroer/taskpyro

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

A service daemon to run Scrapy spiders

Python task scheduler with a user-friendly web UI

A self-hosted Python script automation platform. Upload a script, schedule it, monitor it — nothing else to configure

Stack blocks visually to build complex browser workflows and execute them via API

Tiny web monitoring, scraping & browser automation engine with Lua scripting, built-in storage, dedup, scheduler, hot reload, programmable API server, test runner and desktop/webhook alerts. ~8MB self-contained binary with <10MB idle RAM.

A general-purpose engine for large-scale, repeated information gathering by a fleet of workers.

Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.