iammrduncan/inference-benchmarks

🏎️ Dead-simple LLM benchmarking CLI - latency, cost, and quality metrics

Benchmark your local LLMs with real evaluation suites. Not vibes - actual scores.

Local LLM Inference Speed Test Tool

Provider-agnostic, open-source evaluation infrastructure for language models

Benchmark and optimize LLM inference across frameworks with ease

Comprehensive LLM evaluation at scale: A production-ready framework for evaluating large language models across multiple benchmarks.

Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.

Test LLMs on real tasks. Compare models side-by-side.

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.