AISBench/benchmark

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.

Collection of evals for Inspect AI

Provider-agnostic, open-source evaluation infrastructure for language models

LLM Benchmark for Throughput via Ollama (Local LLMs)

Test LLMs on real tasks. Compare models side-by-side.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends