base on Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. # Promptfoo: LLM evals & red teaming <p align="center"> <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/v/promptfoo" alt="npm"></a> <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/dm/promptfoo" alt="npm"></a> <a href="https://github.com/promptfoo/promptfoo/actions/workflows/main.yml"><img src="https://img.shields.io/github/actions/workflow/status/promptfoo/promptfoo/main.yml" alt="GitHub Workflow Status"></a> <a href="https://github.com/promptfoo/promptfoo/blob/main/LICENSE"><img src="https://img.shields.io/github/license/promptfoo/promptfoo" alt="MIT license"></a> <a href="https://discord.gg/promptfoo"><img src="https://github.com/user-attachments/assets/2092591a-ccc5-42a7-aeb6-24a2808950fd" alt="Discord"></a> </p> <p align="center"> <code>promptfoo</code> is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps. </p> <p align="center"> <a href="https://www.promptfoo.dev">Website</a> · <a href="https://www.promptfoo.dev/docs/getting-started/">Getting Started</a> · <a href="https://www.promptfoo.dev/docs/red-team/">Red Teaming</a> · <a href="https://www.promptfoo.dev/docs/">Documentation</a> · <a href="https://discord.gg/promptfoo">Discord</a> </p> ## Quick Start ```sh # Install and initialize project npx promptfoo@latest init # Run your first evaluation npx promptfoo eval ``` See [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more. ## What can you do with Promptfoo? - **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/) - **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning - **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/)) - **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/) - **Share results** with your team Here's what it looks like in action:  It works on the command line too:  It also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):  ## Why promptfoo? - 🚀 **Developer-first**: Fast, with features like live reload and caching - 🔒 **Private**: Runs 100% locally - your prompts never leave your machine - 🔧 **Flexible**: Works with any LLM API or programming language - 💪 **Battle-tested**: Powers LLM apps serving 10M+ users in production - 📊 **Data-driven**: Make decisions based on metrics, not gut feel - 🤝 **Open source**: MIT licensed, with an active community ## Star the Project ⭐ If you find promptfoo useful, please star it on GitHub! Stars help the project grow and ensure you stay updated on new releases and features. <p align="center"> <img src="site/static/img/github/star-animation.gif" alt="Star us on GitHub!" width="800" /> </p> ## Learn More - 📚 [Full Documentation](https://www.promptfoo.dev/docs/intro/) - 🔐 [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/) - 🎯 [Getting Started](https://www.promptfoo.dev/docs/getting-started/) - 💻 [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/) - 📦 [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/) - 🤖 [Supported Models](https://www.promptfoo.dev/docs/providers/) ## Contributing We welcome contributions! Check out our [contributing guide](https://www.promptfoo.dev/docs/contributing/) to get started. Join our [Discord community](https://discord.gg/promptfoo) for help and discussion. <a href="https://github.com/promptfoo/promptfoo/graphs/contributors"> <img src="https://contrib.rocks/image?repo=promptfoo/promptfoo" /> </a> ", Assign "at most 3 tags" to the expected json: {"id":"965","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"