NVIDIA-NeMo/Evaluator

Evaluate and improve models and agents using environments

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Collection of evals for Inspect AI

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.

Framework for evaluating and improving agents

MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering

Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks.

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.