base on No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents <div align="center"> <img src="docs/assets/unstract_u_logo.png" style="height: 120px"> # Unstract ## Intelligent Document Processing 2.0 (IDP 2.0) Platform Powered by Large Language Models #### No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents ## [](https://pdm-project.org) [](https://cla-assistant.io/Zipstack/unstract) [](https://results.pre-commit.ci/latest/github/Zipstack/unstract/main) [](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) </div> ## 🤖 Prompt Studio Prompt Studio's primary reason for existence is so you can develop the necessary prompts for document data extraction super efficiently. It is a purpose-built environment that makes this not just easy for you—but, lot of fun! The document sample, its variants, the prompts you're developing, outputs from different LLMs, the schema you're developing, costing details of the extraction and various tools that let you measure the effectiveness of your prompts are just a click away and easily accessible. Prompt Studio is designed for effective and high speed development and iteration of prompts for document data extraction. Welcome to IDP 2.0!  ## 🧘‍♀️ Three step nirvana with Workflow Studio Automate critical business processes that involve complex documents with a human in the loop. Go beyond RPA with the power of Large Language Models. 🌟 **Step 1**: Add documents to no-code Prompt Studio and do prompt engineering to extract required fields <br> 🌟 **Step 2**: Configure Prompt Studio project as API deployment or configure input source and output destination for ETL Pipeline<br> 🌟 **Step 3**: Deploy Workflows as unstructured data APIs or unstructured data ETL Pipelines!  ## 🚀 Getting started ### System Requirements - 8GB RAM (recommended) ### Prerequisites - Linux or MacOS (Intel or M-series) - Docker - Docker Compose (if you need to install it separately) - Git Next, either download a release or clone this repo and do the following: ✅ `./run-platform.sh`<br> ✅ Now visit [http://frontend.unstract.localhost](http://frontend.unstract.localhost) in your browser <br> ✅ Use username and password `unstract` to login That's all there is to it! Follow [these steps](backend/README.md#authentication) to change the default username and password. See [user guide](https://docs.unstract.com/unstract/unstract_platform/user_guides/run_platform) for more details on managing the platform. Another really quick way to experience Unstract is by signing up for our [hosted version](https://us-central.unstract.com/). It comes with a 14 day free trial! ## ⏩ Quick Start Guide Unstract comes well documented. You can get introduced to the [basics of Unstract](https://docs.unstract.com/unstract/), and [learn how to connect](https://docs.unstract.com/unstract/unstract_platform/setup_accounts/whats_needed) various systems like LLMs, Vector Databases, Embedding Models and Text Extractors to it. The easiest way to wet your feet is to go through our [Quick Start Guide](https://docs.unstract.com/unstract/unstract_platform/quick_start) where you actually get to do some prompt engineering in Prompt Studio and launch an API to structure varied credit card statements! ## 🤝 Ecosystem support ### LLM Providers || Provider | Status | |----------------------------------------------------------------|-----------------------------|---| | <img src="docs/assets/3rd_party/openai.png" width="32"/> | OpenAI | ✅ Working | | <img src="docs/assets/3rd_party/vertex_ai.png" width="32"/> | Google VertexAI, Gemini Pro | ✅ Working | | <img src="docs/assets/3rd_party/azure_openai.png" width="32"/> | Azure OpenAI | ✅ Working | | <img src="docs/assets/3rd_party/anthropic.png" width="32"/> | Anthropic | ✅ Working | | <img src="docs/assets/3rd_party/ollama.png" width="32"/> | Ollama | ✅ Working | | <img src="docs/assets/3rd_party/bedrock.png" width="32"/> | Bedrock | ✅ Working | | <img src="docs/assets/3rd_party/palm.png" width="32"/> | Google PaLM | ✅ Working | | <img src="docs/assets/3rd_party/anyscale.png" width="32"/> | Anyscale | ✅ Working | | <img src="docs/assets/3rd_party/mistral_ai.png" width="32"/> | Mistral AI | ✅ Working | | <img src="docs/assets/3rd_party/replicate.png" width="32"/> | Replicate | 🗓️ Coming soon! | ### Vector Databases || Provider | Status | |---|---|---| |<img src="docs/assets/3rd_party/qdrant.png" width="32"/>| Qdrant | ✅ Working | |<img src="docs/assets/3rd_party/weaviate.png" width="32"/>| Weaviate | ✅ Working | |<img src="docs/assets/3rd_party/pinecone.png" width="32"/>| Pinecone | ✅ Working | |<img src="docs/assets/3rd_party/postgres.png" width="32"/>| PostgreSQL | ✅ Working | |<img src="docs/assets/3rd_party/milvus.png" width="32"/>| Milvus | ✅ Working | ### Embeddings || Provider | Status | |---|---|---| |<img src="docs/assets/3rd_party/openai.png" width="32"/>| OpenAI | ✅ Working | |<img src="docs/assets/3rd_party/azure_openai.png" width="32"/>| Azure OpenAI | ✅ Working | |<img src="docs/assets/3rd_party/palm.png" width="32"/>| Google PaLM | ✅ Working | |<img src="docs/assets/3rd_party/ollama.png" width="32"/>| Ollama | ✅ Working | ### Text Extractors || Provider | Status | |---|---|---| |<img src="docs/assets/unstract_u_logo.png" width="32"/>| Unstract LLMWhisperer | ✅ Working | |<img src="docs/assets/3rd_party/unstructured_io.png" width="32"/>| Unstructured.io Community | ✅ Working | |<img src="docs/assets/3rd_party/unstructured_io.png" width="32"/>| Unstructured.io Enterprise | ✅ Working | |<img src="docs/assets/3rd_party/llamaindex.png" width="32"/>| LlamaIndex Parse | ✅ Working | ### ETL Sources || Provider | Status | |---|---|---| |<img src="docs/assets/3rd_party/s3.png" width="32"/>| AWS S3 | ✅ Working | |<img src="docs/assets/3rd_party/minio.png" width="32"/>| MinIO | ✅ Working | |<img src="docs/assets/3rd_party/gcp.png" width="32"/>| Google Cloud Storage | ✅ Working | |<img src="docs/assets/3rd_party/azure_openai.png" width="32"/>| Azure Cloud Storage | ✅ Working | |<img src="docs/assets/3rd_party/google_drive.png" width="32"/>| Google Drive | ✅ Working | |<img src="docs/assets/3rd_party/dropbox.png" width="32"/>| Dropbox | ✅ Working | |<img src="docs/assets/3rd_party/sftp.png" width="32"/>| SFTP | ✅ Working | |<img src="docs/assets/3rd_party/box.png" width="32"/>| Box | 🗓️ Coming soon! | |<img src="docs/assets/3rd_party/http.png" width="32"/>| HTTP/HTTPS | 🗓️ Coming soon! | ### ETL Destinations || Provider | Status | |---|---|---| |<img src="docs/assets/3rd_party/snowflake.png" width="32"/>| Snowflake | ✅ Working | |<img src="docs/assets/3rd_party/amazon_redshift.png" width="32"/>| Amazon Redshift | ✅ Working | |<img src="docs/assets/3rd_party/google_bigquery.png" width="32"/>| Google BigQuery | ✅ Working | |<img src="docs/assets/3rd_party/postgres.png" width="32"/>| PostgreSQL | ✅ Working | |<img src="docs/assets/3rd_party/mysql.png" width="32"/>| MySQL | ✅ Working | |<img src="docs/assets/3rd_party/mariadb.png" width="32"/>| MariaDB | ✅ Working | |<img src="docs/assets/3rd_party/ms_sql.png" width="32"/>| Microsoft SQL Server | ✅ Working | ## 🙌 Contributing Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for further details to get started easily. ## 👋 Join the LLM-powered automation community - On Slack, [join great conversations](https://join-slack.unstract.com) around LLMs, their ecosystem and leveraging them to automate the previously unautomatable! - [Follow us on X/Twitter](https://twitter.com/GetUnstract) - [Follow us on LinkedIn](https://www.linkedin.com/showcase/unstract/) ## 🚨 Backup encryption key Do copy the value of `ENCRYPTION_KEY` config in either `backend/.env` or `platform-service/.env` file to a secure location. Adapter credentials are encrypted by the platform using this key. Its loss or change will make all existing adapters inaccessible! ## 📊 A note on analytics In full disclosure, Unstract integrates Posthog to track usage analytics. As you can inspect the relevant code here, we collect the minimum possible metrics. Posthog can be disabled if desired by setting `REACT_APP_ENABLE_POSTHOG` to `false` in the frontend's .env file. ", Assign "at most 3 tags" to the expected json: {"id":"11465","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"