base on An LLM playground you can run on your laptop # openplayground An LLM playground you can run on your laptop. https://user-images.githubusercontent.com/111631/227399583-39b23f48-9823-4571-a906-985dbe282b20.mp4 #### Features - Use any model from [OpenAI](https://openai.com), [Anthropic](https://anthropic.com), [Cohere](https://cohere.com), [Forefront](https://forefront.ai), [HuggingFace](https://huggingface.co), [Aleph Alpha](https://aleph-alpha.com), [Replicate](https://replicate.com), [Banana](https://banana.dev) and [llama.cpp](https://github.com/ggerganov/llama.cpp). - Full playground UI, including history, parameter tuning, keyboard shortcuts, and logprops. - Compare models side-by-side with the same prompt, individually tune model parameters, and retry with different parameters. - Automatically detects local models in your HuggingFace cache, and lets you install new ones. - Works OK on your phone. - Probably won't kill everyone. ## Try on nat.dev Try the hosted version: [nat.dev](https://nat.dev). ## How to install and run ```sh pip install openplayground openplayground run ``` Alternatively, run it as a docker container: ```sh docker run --name openplayground -p 5432:5432 -d --volume openplayground:/web/config natorg/openplayground ``` This runs a Flask process, so you can add the typical flags such as setting a different port `openplayground run -p 1235` and others. ## How to run for development ```sh git clone https://github.com/nat/openplayground cd app && npm install && npx parcel watch src/index.html --no-cache cd server && pip3 install -r requirements.txt && cd .. && python3 -m server.app ``` ## Docker ```sh docker build . --tag "openplayground" docker run --name openplayground -p 5432:5432 -d --volume openplayground:/web/config openplayground ``` First volume is optional. It's used to store API keys, models settings. ## Ideas for contributions - Add a token counter to the playground - Add a cost counter to the playground and the compare page - Measure and display time to first token - Setup automatic builds with GitHub Actions - The default parameters for each model are configured in the `server/models.json` file. If you find better default parameters for a model, please submit a pull request! - Someone can help us make a homebrew package, and a dockerfile - Easier way to install open source models directly from openplayground, with `openplayground install <model>` or in the UI. - Find and fix bugs - ChatGPT UI, with turn-by-turn, markdown rendering, chatgpt plugin support, etc. - We will probably need multimodal inputs and outputs at some point in 2023 ### llama.cpp ## Adding models to openplayground Models and providers have three types in openplayground: - Searchable - Local inference - API You can add models in `server/models.json` with the following schema: #### Local inference For models running locally on your device you can add them to openplayground like the following (a minimal example): ```json "llama": { "api_key" : false, "models" : { "llama-70b": { "parameters": { "temperature": { "value": 0.5, "range": [ 0.1, 1.0 ] }, } } } } ``` Keep in mind you will need to add a generation method for your model in `server/app.py`. Take a look at `local_text_generation()` as an example. #### API Provider Inference This is for model providers like OpenAI, cohere, forefront, and more. You can connect them easily into openplayground (a minimal example): ```json "cohere": { "api_key" : true, "models" : { "xlarge": { "parameters": { "temperature": { "value": 0.5, "range": [ 0.1, 1.0 ] }, } } } } ``` Keep in mind you will need to add a generation method for your model in `server/app.py`. Take a look at `openai_text_generation()` or `cohere_text_generation()` as an example. #### Searchable models We use this for Huggingface Remote Inference models, the search endpoint is useful for scaling to N models in the settings page. ```json "provider_name": { "api_key": true, "search": { "endpoint": "ENDPOINT_URL" }, "parameters": { "parameter": { "value": 1.0, "range": [ 0.1, 1.0 ] }, } } ``` #### Credits Instigated by Nat Friedman. Initial implementation by [Zain Huda](https://github.com/zainhuda) as a repl.it bounty. Many features and extensive refactoring by [Alex Lourenco](https://github.com/AlexanderLourenco). ", Assign "at most 3 tags" to the expected json: {"id":"2391","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"