base on Extract clean data from anywhere, powered by vision-language models ⚡ <div align="center"> <a href="https://thepi.pe/"> <img src="https://rpnutzemutbrumczwvue.supabase.co/storage/v1/object/public/assets/pipeline_small%20(1).png" alt="Pipeline Illustration" style="width:96px; height:72px; vertical-align:middle;"> <h1>thepi.pe</h1> </a> <a> <img src="https://github.com/emcf/thepipe/actions/workflows/python-ci.yml/badge.svg" alt="python-gh-action"> </a> <a href="https://codecov.io/gh/emcf/thepipe"> <img src="https://codecov.io/gh/emcf/thepipe/graph/badge.svg?token=OE7CUEFUL9" alt="codecov"> </a> <a href="https://raw.githubusercontent.com/emcf/thepipe/main/LICENSE"> <img src="https://img.shields.io/badge/license-MIT-green" alt="MIT license"> </a> <a href="https://www.pepy.tech/projects/thepipe-api"> <img src="https://static.pepy.tech/badge/thepipe-api" alt="PyPI"> </a> <a href="https://thepi.pe/"> <img src="https://img.shields.io/website?url=https%3A%2F%2Fthepipe-api.up.railway.app%2F&label=API%20status" alt="Website"> </a> </div> ### Extract clean data from tricky documents ⚡ thepi.pe is a package that can scrape clean markdown or accurately extract structured data from complex documents. It uses vision-language models (VLMs) under the hood, and works out-of-the-box with any LLM, VLM, or vector database. It can be used right away on a [hosted cloud](https://thepi.pe), or it can be run locally. ## Features 🌟 - Scrape clean markdown, tables, and images from any document or webpage - Works out-of-the-box with LLMs, vector databases, and RAG frameworks - AI-native filetype detection, layout analysis, and structured data extraction - Accepts a wide range of sources, including PDFs, URLs, Word docs, Powerpoints, Python notebooks, GitHub repos, videos, audio, and more ## Get started in 5 minutes 🚀 thepi.pe can read a wide range of filetypes and web sources, so it requires a few dependencies. It also requires vision-language model inference for AI extraction features. For these reasons, we host an API that works out-of-the-box. For more detailed setup instructions, view the [docs](https://thepi.pe/docs-platform). ```bash pip install thepipe-api ``` ### Hosted API (Python) You can get an API key by signing up for a free account at [thepi.pe](https://thepi.pe). It is completely free to try out. The, simply set the `THEPIPE_API_KEY` environment variable to your API key. ## Scrape Function ```python from thepipe.scraper import scrape_file from thepipe.core import chunks_to_messages from openai import OpenAI # scrape clean markdown chunks = scrape_file(filepath="paper.pdf", ai_extraction=False) # call LLM with scraped chunks client = OpenAI() response = client.chat.completions.create( model="gpt-4o", messages=chunks_to_messages(chunks), ) ``` The output from thepi.pe is a list of chunks containing all content within the source document. These chunks can easily be converted to a prompt format that is compatible with any LLM or multimodal model with `thepipe.core.chunks_to_messages`, which gives the following format: ```json [ { "role": "user", "content": [ { "type": "text", "text": "..." }, { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,..." } } ] } ] ``` ## Extract Function The extract function allows you to extract structured data from documents. You can use it as follows: ```python from thepipe.extract import extract_from_file # Define your schema schema = { "name": "string", "age": "int", "is_student": "bool" } # Extract data from the file result = extract_from_file( file_path="document.pdf", schema=schema, ai_model="gpt-4o-mini", multiple_extractions=True ) print(result) ``` ### Local Installation (Python) For a local installation, you can use the following command: ```bash pip install thepipe-api[local] ``` You must have a local LLM server setup and running for AI extraction features. You can use any local LLM server that follows OpenAI format (such as [LiteLLM](https://github.com/BerriAI/litellm)) or a provider (such as [OpenRouter](https://openrouter.ai/) or [OpenAI](https://platform.openai.com/)). Next, set the `LLM_SERVER_BASE_URL` environment variable to your LLM server's endpoint URL and set `LLM_SERVER_API_KEY`. the `DEFAULT_AI_MODEL` environment variable can be set to your VLM of choice. For example, you would use `openai/gpt-4o-mini` if using OpenRouter or `gpt-4o-mini` if using OpenAI. For full functionality with media-rich sources, you will need to install the following dependencies: ```bash apt-get update && apt-get install -y git ffmpeg tesseract-ocr python -m playwright install --with-deps chromium ``` When using thepi.pe locally, be sure to append `local=True` to your function calls: ```python chunks = scrape_url(url="https://example.com", local=True) ``` You can also use thepi.pe from the command line: ```bash thepipe path/to/folder --include_regex .*\.tsx --local ``` ## Supported File Types 📚 | Source | Input types | Multimodal | Notes | |--------------------------|----------------------------------------------------------------|---------------------|----------------------| | Webpage | URLs starting with `http`, `https`, `ftp` | ✔️ | Scrapes markdown, images, and tables from web pages. `ai_extraction` available for AI content extraction from the webpage's screenshot | | PDF | `.pdf` | ✔️ | Extracts page markdown and page images. `ai_extraction` available to use a VLM for complex or scanned documents | | Word Document | `.docx` | ✔️ | Extracts text, tables, and images | | PowerPoint | `.pptx` | ✔️ | Extracts text and images from slides | | Video | `.mp4`, `.mov`, `.wmv` | ✔️ | Uses Whisper for transcription and extracts frames | | Audio | `.mp3`, `.wav` | ✔️ | Uses Whisper for transcription | | Jupyter Notebook | `.ipynb` | ✔️ | Extracts markdown, code, outputs, and images | | Spreadsheet | `.csv`, `.xls`, `.xlsx` | ❌ | Converts each row to JSON format, including row index for each | | Plaintext | `.txt`, `.md`, `.rtf`, etc | ❌ | Simple text extraction | | Image | `.jpg`, `.jpeg`, `.png` | ✔️ | Uses pytesseract for OCR in text-only mode | | ZIP File | `.zip` | ✔️ | Extracts and processes contained files | | Directory | any `path/to/folder` | ✔️ | Recursively processes all files in directory | | YouTube Video (known issues) | YouTube video URLs starting with `https://youtube.com` or `https://www.youtube.com`. | ✔️ | Uses pytube for video download and Whisper for transcription. For consistent extraction, you may need to modify your `pytube` installation to send a valid user agent header (see [this issue](https://github.com/pytube/pytube/issues/399)). | | Tweet | URLs starting with `https://twitter.com` or `https://x.com` | ✔️ | Uses unofficial API, may break unexpectedly | | GitHub Repository | GitHub repo URLs starting with `https://github.com` or `https://www.github.com` | ✔️ | Requires GITHUB_TOKEN environment variable | ## How it works 🛠️ thepi.pe uses computer vision models and heuristics to extract clean content from the source and process it for downstream use with [language models](https://en.wikipedia.org/wiki/Large_language_model), or [vision transformers](https://en.wikipedia.org/wiki/Vision_transformer). You can feed these messages directly into the model, or alternatively you can use `chunker.chunk_by_document`, `chunker.chunk_by_page`, `chunker.chunk_by_section`, `chunker.chunk_semantic` to chunk these messages for a vector database such as ChromaDB or a RAG framework. A chunk can be converted to LlamaIndex Document/ImageDocument with `.to_llamaindex`. > ⚠️ **It is important to be mindful of your model's token limit.** GPT-4o does not work with too many images in the prompt (see discussion [here](https://community.openai.com/t/gpt-4-vision-maximum-amount-of-images/573110/6)). To remedy this issue, either use an LLM with a larger context window, extract larger documents with `text_only=True`, or embed the chunks into vector database. # Sponsors <a href="https://cal.com/emmett-mcf/30min"><img alt="Book us with Cal.com" src="https://cal.com/book-with-cal-dark.svg" /></a> Thank you to [Cal.com](https://cal.com/) for sponsoring this project.", Assign "at most 3 tags" to the expected json: {"id":"8905","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"