base on An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic. # Open Deep Research An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction over time and deep dive into a topic. Goal is to keep the repo size at <500 LoC so it is easy to understand and build on top of. If you like this project, please consider starring it and giving me a follow on [X/Twitter](https://x.com/dzhng). This project is sponsored by [Aomni](https://aomni.com). ## How It Works ```mermaid flowchart TB subgraph Input Q[User Query] B[Breadth Parameter] D[Depth Parameter] end DR[Deep Research] --> SQ[SERP Queries] --> PR[Process Results] subgraph Results[Results] direction TB NL((Learnings)) ND((Directions)) end PR --> NL PR --> ND DP{depth > 0?} RD["Next Direction: - Prior Goals - New Questions - Learnings"] MR[Markdown Report] %% Main Flow Q & B & D --> DR %% Results to Decision NL & ND --> DP %% Circular Flow DP -->|Yes| RD RD -->|New Context| DR %% Final Output DP -->|No| MR %% Styling classDef input fill:#7bed9f,stroke:#2ed573,color:black classDef process fill:#70a1ff,stroke:#1e90ff,color:black classDef recursive fill:#ffa502,stroke:#ff7f50,color:black classDef output fill:#ff4757,stroke:#ff6b81,color:black classDef results fill:#a8e6cf,stroke:#3b7a57,color:black class Q,B,D input class DR,SQ,PR process class DP,RD recursive class MR output class NL,ND results ``` ## Features - **Iterative Research**: Performs deep research by iteratively generating search queries, processing results, and diving deeper based on findings - **Intelligent Query Generation**: Uses LLMs to generate targeted search queries based on research goals and previous findings - **Depth & Breadth Control**: Configurable parameters to control how wide (breadth) and deep (depth) the research goes - **Smart Follow-up**: Generates follow-up questions to better understand research needs - **Comprehensive Reports**: Produces detailed markdown reports with findings and sources - **Concurrent Processing**: Handles multiple searches and result processing in parallel for efficiency ## Requirements - Node.js environment - API keys for: - Firecrawl API (for web search and content extraction) - OpenAI API (for o3 mini model) ## Setup ### Node.js 1. Clone the repository 2. Install dependencies: ```bash npm install ``` 3. Set up environment variables in a `.env.local` file: ```bash FIRECRAWL_KEY="your_firecrawl_key" # If you want to use your self-hosted Firecrawl, add the following below: # FIRECRAWL_BASE_URL="http://localhost:3002" OPENAI_KEY="your_openai_key" ``` To use local LLM, comment out `OPENAI_KEY` and instead uncomment `OPENAI_ENDPOINT` and `OPENAI_MODEL`: - Set `OPENAI_ENDPOINT` to the address of your local server (eg."http://localhost:1234/v1") - Set `OPENAI_MODEL` to the name of the model loaded in your local server. ### Docker 1. Clone the repository 2. Rename `.env.example` to `.env.local` and set your API keys 3. Run `docker build -f Dockerfile` 4. Run the Docker image: ```bash docker compose up -d ``` 5. Execute `npm run docker` in the docker service: ```bash docker exec -it deep-research npm run docker ``` ## Usage Run the research assistant: ```bash npm start ``` You'll be prompted to: 1. Enter your research query 2. Specify research breadth (recommended: 3-10, default: 4) 3. Specify research depth (recommended: 1-5, default: 2) 4. Answer follow-up questions to refine the research direction The system will then: 1. Generate and execute search queries 2. Process and analyze search results 3. Recursively explore deeper based on findings 4. Generate a comprehensive markdown report The final report will be saved as `report.md` or `answer.md` in your working directory, depending on which modes you selected. ### Concurrency If you have a paid version of Firecrawl or a local version, feel free to increase the `ConcurrencyLimit` by setting the `CONCURRENCY_LIMIT` environment variable so it runs faster. If you have a free version, you may sometimes run into rate limit errors, you can reduce the limit to 1 (but it will run a lot slower). ### DeepSeek R1 Deep research performs great on R1! We use [Fireworks](http://fireworks.ai) as the main provider for the R1 model. To use R1, simply set a Fireworks API key: ```bash FIREWORKS_KEY="api_key" ``` The system will automatically switch over to use R1 instead of `o3-mini` when the key is detected. ### Custom endpoints and models There are 2 other optional env vars that lets you tweak the endpoint (for other OpenAI compatible APIs like OpenRouter or Gemini) as well as the model string. ```bash OPENAI_ENDPOINT="custom_endpoint" CUSTOM_MODEL="custom_model" ``` ## How It Works 1. **Initial Setup** - Takes user query and research parameters (breadth & depth) - Generates follow-up questions to understand research needs better 2. **Deep Research Process** - Generates multiple SERP queries based on research goals - Processes search results to extract key learnings - Generates follow-up research directions 3. **Recursive Exploration** - If depth > 0, takes new research directions and continues exploration - Each iteration builds on previous learnings - Maintains context of research goals and findings 4. **Report Generation** - Compiles all findings into a comprehensive markdown report - Includes all sources and references - Organizes information in a clear, readable format ## License MIT License - feel free to use and modify as needed. ", Assign "at most 3 tags" to the expected json: {"id":"13691","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"