modelscope/evalscope

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

Inspect: A framework for large language model evaluations

The LLM Evaluation Framework

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

A unified evaluation framework for large language models

Supercharge Your LLM Application Evaluations 🚀

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.