flashinfer-ai/flashinfer-bench

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

llama-benchy - llama-bench style benchmarking tool for all backends

KernelBench: Can LLMs Write GPU Kernels? - Benchmark + Toolkit with Torch -> CUDA (+ more DSLs)

FlashInfer: Kernel Library for LLM Serving

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

LiveBench: A Challenging, Contamination-Free LLM Benchmark