base on # OpenAI Realtime API with Twilio Quickstart Combine OpenAI's Realtime API and Twilio's phone calling capability to build an AI calling assistant. <img width="1728" alt="Screenshot 2024-12-18 at 4 59 30 PM" src="https://github.com/user-attachments/assets/d3c8dcce-b339-410c-85ca-864a8e0fc326" /> ## Quick Setup Open three terminal windows: | Terminal | Purpose | Quick Reference (see below for more) | | -------- | ----------------------------- | ------------------------------------ | | 1 | To run the `webapp` | `npm run dev` | | 2 | To run the `websocket-server` | `npm run dev` | | 3 | To run `ngrok` | `ngrok http 8081` | Make sure all vars in `webapp/.env` and `websocket-server/.env` are set correctly. See [full setup](#full-setup) section for more. ## Overview This repo implements a phone calling assistant with the Realtime API and Twilio, and had two main parts: the `webapp`, and the `websocket-server`. 1. `webapp`: NextJS app to serve as a frontend for call configuration and transcripts 2. `websocket-server`: Express backend that handles connection from Twilio, connects it to the Realtime API, and forwards messages to the frontend <img width="1514" alt="Screenshot 2024-12-20 at 10 32 40 AM" src="https://github.com/user-attachments/assets/61d39b88-4861-4b6f-bfe2-796957ab5476" /> Twilio uses TwiML (a form of XML) to specify how to handle a phone call. When a call comes in we tell Twilio to start a bi-directional stream to our backend, where we forward messages between the call and the Realtime API. (`{{WS_URL}}` is replaced with our websocket endpoint.) ```xml <!-- TwiML to start a bi-directional stream--> <?xml version="1.0" encoding="UTF-8"?> <Response> <Say>Connected</Say> <Connect> <Stream url="{{WS_URL}}" /> </Connect> <Say>Disconnected</Say> </Response> ``` We use `ngrok` to make our server reachable by Twilio. ### Life of a phone call Setup 1. We run ngrok to make our server reachable by Twilio 1. We set the Twilio webhook to our ngrok address 1. Frontend connects to the backend (`wss://[your_backend]/logs`), ready for a call Call 1. Call is placed to Twilio-managed number 1. Twilio queries the webhook (`http://[your_backend]/twiml`) for TwiML instructions 1. Twilio opens a bi-directional stream to the backend (`wss://[your_backend]/call`) 1. The backend connects to the Realtime API, and starts forwarding messages: - between Twilio and the Realtime API - between the frontend and the Realtime API ### Function Calling This demo mocks out function calls so you can provide sample responses. In reality you could handle the function call, execute some code, and then supply the response back to the model. ## Full Setup 1. Make sure your [auth & env](#detailed-auth--env) is configured correctly. 2. Run webapp. ```shell cd webapp npm install npm run dev ``` 3. Run websocket server. ```shell cd websocket-server npm install npm run dev ``` ## Detailed Auth & Env ### OpenAI & Twilio Set your credentials in `webapp/.env` and `websocket-server` - see `webapp/.env.example` and `websocket-server.env.example` for reference. ### Ngrok Twilio needs to be able to reach your websocket server. If you're running it locally, your ports are inaccessible by default. [ngrok](https://ngrok.com/) can make them temporarily accessible. We have set the `websocket-server` to run on port `8081` by default, so that is the port we will be forwarding. ```shell ngrok http 8081 ``` Make note of the `Forwarding` URL. (e.g. `https://54c5-35-170-32-42.ngrok-free.app`) ### Websocket URL Your server should now be accessible at the `Forwarding` URL when run, so set the `PUBLIC_URL` in `websocket-server/.env`. See `websocket-server/.env.example` for reference. # Additional Notes This repo isn't polished, and the security practices leave some to be desired. Please only use this as reference, and make sure to audit your app with security and engineering before deploying! ", Assign "at most 3 tags" to the expected json: {"id":"14705","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"