base on Open Source Alternative to NotebookLM / Perplexity / Glean, connected to external sources such as search engines (Tavily), Slack, Linear, Notion, YouTube, GitHub and more.  # SurfSense While tools like NotebookLM and Perplexity are impressive and highly effective for conducting research on any topic/query, SurfSense elevates this capability by integrating with your personal knowledge base. It is a highly customizable AI research agent, connected to external sources such as search engines (Tavily, LinkUp), Slack, Linear, Notion, YouTube, GitHub and more to come. # Video https://github.com/user-attachments/assets/48142909-6391-4084-b7e8-81da388bb1fc ## Key Features ### 1. Latest #### 💡 **Idea**: Have your own highly customizable private NotebookLM and Perplexity integrated with external sources. #### 📁 **Multiple File Format Uploading Support** Save content from your own personal files *(Documents, images and supports **27 file extensions**)* to your own personal knowledge base . #### 🔍 **Powerful Search** Quickly research or find anything in your saved content . #### 💬 **Chat with your Saved Content** Interact in Natural Language and get cited answers. #### 📄 **Cited Answers** Get Cited answers just like Perplexity. #### 🔔 **Privacy & Local LLM Support** Works Flawlessly with Ollama local LLMs. #### 🏠 **Self Hostable** Open source and easy to deploy locally. #### 📊 **Advanced RAG Techniques** - Supports 150+ LLM's - Supports 6000+ Embedding Models. - Supports all major Rerankers (Pinecode, Cohere, Flashrank etc) - Uses Hierarchical Indices (2 tiered RAG setup). - Utilizes Hybrid Search (Semantic + Full Text Search combined with Reciprocal Rank Fusion). - RAG as a Service API Backend. #### ℹ️ **External Sources** - Search Engines (Tavily, LinkUp) - Slack - Linear - Notion - Youtube Videos - GitHub - and more to come..... #### 🔖 Cross Browser Extension - The SurfSense extension can be used to save any webpage you like. - Its main usecase is to save any webpages protected beyond authentication. ### 2. Temporarily Deprecated #### Podcasts - The SurfSense Podcast feature is currently being reworked for better UI and stability. Expect it soon. ## FEATURE REQUESTS AND FUTURE **SurfSense is actively being developed.** While it's not yet production-ready, you can help us speed up the process. Join the [SurfSense Discord](https://discord.gg/ejRNvftDp9) and help shape the future of SurfSense! ## How to get started? ### Installation Options SurfSense provides two installation methods: 1. **[Docker Installation](https://www.surfsense.net/docs/docker-installation)** - The easiest way to get SurfSense up and running with all dependencies containerized. Less Customization. 2. **[Manual Installation (Recommended)](https://www.surfsense.net/docs/manual-installation)** - For users who prefer more control over their setup or need to customize their deployment. Both installation guides include detailed OS-specific instructions for Windows, macOS, and Linux. Before installation, make sure to complete the [prerequisite setup steps](https://www.surfsense.net/docs/) including: - PGVector setup - Google OAuth configuration - Unstructured.io API key - Other required API keys ## Screenshots **Search Spaces**  **Manage Documents**  **Research Agent**  **Agent Chat**  **Browser Extension**   ## Tech Stack ### **BackEnd** - **FastAPI**: Modern, fast web framework for building APIs with Python - **PostgreSQL with pgvector**: Database with vector search capabilities for similarity searches - **SQLAlchemy**: SQL toolkit and ORM (Object-Relational Mapping) for database interactions - **Alembic**: A database migrations tool for SQLAlchemy. - **FastAPI Users**: Authentication and user management with JWT and OAuth support - **LangGraph**: Framework for developing AI-agents. - **LangChain**: Framework for developing AI-powered applications. - **LLM Integration**: Integration with LLM models through LiteLLM - **Rerankers**: Advanced result ranking for improved search relevance - **Hybrid Search**: Combines vector similarity and full-text search for optimal results using Reciprocal Rank Fusion (RRF) - **Vector Embeddings**: Document and text embeddings for semantic search - **pgvector**: PostgreSQL extension for efficient vector similarity operations - **Chonkie**: Advanced document chunking and embedding library - Uses `AutoEmbeddings` for flexible embedding model selection - `LateChunker` for optimized document chunking based on embedding model's max sequence length --- ### **FrontEnd** - **Next.js 15.2.3**: React framework featuring App Router, server components, automatic code-splitting, and optimized rendering. - **React 19.0.0**: JavaScript library for building user interfaces. - **TypeScript**: Static type-checking for JavaScript, enhancing code quality and developer experience. - **Vercel AI SDK Kit UI Stream Protocol**: To create scalable chat UI. - **Tailwind CSS 4.x**: Utility-first CSS framework for building custom UI designs. - **Shadcn**: Headless components library. - **Lucide React**: Icon set implemented as React components. - **Framer Motion**: Animation library for React. - **Sonner**: Toast notification library. - **Geist**: Font family from Vercel. - **React Hook Form**: Form state management and validation. - **Zod**: TypeScript-first schema validation with static type inference. - **@hookform/resolvers**: Resolvers for using validation libraries with React Hook Form. - **@tanstack/react-table**: Headless UI for building powerful tables & datagrids. ### **Extension** Manifest v3 on Plasmo ## Future Work - Add More Connectors. - Patch minor bugs. - Implement Canvas. - Complete Hybrid Search. **[Done]** - Add support for file uploads QA. **[Done]** - Shift to WebSockets for Streaming responses. **[Deprecated in favor of AI SDK Stream Protocol]** - Based on feedback, I will work on making it compatible with local models. **[Done]** - Cross Browser Extension **[Done]** - Critical Notifications **[Done | PAUSED]** - Saving Chats **[Done]** - Basic keyword search page for saved sessions **[Done]** - Multi & Single Document Chat **[Done]** ## Contribute Contributions are very welcome! A contribution can be as small as a ⭐ or even finding and creating issues. Fine-tuning the Backend is always desired. ", Assign "at most 3 tags" to the expected json: {"id":"13606","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"