base on # AI Video Search Engine (AVSE) ## A video search engine powered by the latest tools in AI ## Why? With the rise of short form content with TikTok and Youtube. A lot more knowledge is in videos than ever before. Finding specific answers within millions of videos can be difficult for any one person to go through. So the question is if there is Google that indexes text on website making it easier to find based on the context of on your question, why is there no Google that indexes video content making it easier for users to find answers within them. So I built this to showcase that it's very much possible with the technology and infrastructure that is readily available. ## Tech Stack - Supbase (PostgreSQL, PG_Vector, Auth) - Hasura (GraphQL layer, permissions) - Fly (Hosting of Hasura) - JigsawStack (Summary AI, Chat AI) - Vercel (NextJS hosting, Serverless functions) ## How it works? ### Storing of videos - Video transcription is extracted from youtube video - Transcriptions is chunked along with timestamp based on https://huggingface.co/Supabase/gte-small dimension size - Stored in postgres DB with use of pg_vector extension and indexing ### Searching - Vector cosine search across the db based on question to return the relavant results - Each video in the result will perform a second search to find related chunks of video to the question - The chunks will be mapped back to the timestamped transcript to play the relavant clip of the video ### Summary & Chat - Transcription of video will be sent to JigsawStack API for summary in both point form and text - Chat sessions will be created and managed by JigsawStack API, related chunks of the video will be sent to chat session based on questions ## Things to note to host it yourself - You'll need a paid Supabase & Fly.io account if you're planning to index thousands to millions of videos - `admin/config/fly.toml` Consist of configs needed to deploy Hasura to fly - `admin/migration` Migration dump you can use to recreate the schema through Hasura CLI - `hasura init migration --endpoint <hasuraurl.fly.app> --admin-secret <admin_secret>` To update migration folder in `admin` - `admin/indexChannelVideos.ts` Script to index large number of videos locally with youtube channels - `.env.example` Keys needed to run to project ## FAQ Doesn't youtube do this? - Not really, Youtube doesn't search the transcribed audio of the video but instead relies on the written content of the uploader such as title, description, tags. While all the audio content goes unindexed. How's this stack gonna handle millions of videos? - It can handle millions but maybe not billions/trillions with this current setup. Which requires more replicas, instances and especially $ ## What's next? - Add TikTok as a video source - Add https://replicate.com/vaibhavs10/incredibly-fast-whisper to transcribe audio - Improve query performance significantly - Page to view all active chats", Assign "at most 3 tags" to the expected json: {"id":"6296","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"