typesafe-ai/system-one-adapter-python

Readymade evaluators for your LLM apps

Evaluate your LLM's response with Prometheus and GPT4 💯

Inspect: A framework for large language model evaluations

The Universe of Evaluation. All about the evaluation for LLMs.

OpenAI Guardrails - Python

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Supercharge Your LLM Application Evaluations 🚀

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.