aisa-group/InferenceBench

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Evaluating frontier models on real-world ai-infra engineering tasks

Benchmark and optimize LLM inference across frameworks with ease

A Long-Horizon Benchmark for Self-Improving Inference Systems

Building the Virtuous Cycle for AI-driven LLM Systems

Phi-Bench (Φ-Bench): 85 open-source LLM-infrastructure engineering tasks for frontier LLMs & coding agents (KFC/LH/E2E). Self-contained public Dockerfile + offline scoring + reference solution per task. Website: llminfrabench.com

📊 LLM Context Benchmarks - A comprehensive benchmarking tool for testing LLMs with varying context sizes using Ollama. Features dual benchmark modes (API/CLI), automatic hardware detection (optimized for Apple Silicon), visual performance charts.

Measuring and evolving with the frontier of agent work

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.