facebookresearch/llm-speedrunner

LLM Arena by KCORES team

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Reproducible Language Agent Research

Taste-Bench: measuring the long-horizon judgment of LLM agents at real decision forks

AIRS-Bench: an AI Research Science benchmark for quantifying the end-to-end AI research abilities of LLM agents

A benchmark to evaluate language models on questions I've previously asked them to solve.

Phi-Bench (Φ-Bench): 85 open-source LLM-infrastructure engineering tasks for frontier LLMs & coding agents (KFC/LH/E2E). Self-contained public Dockerfile + offline scoring + reference solution per task. Website: llminfrabench.com