DogukanUrker/BenchKit

Test LLMs on real tasks. Compare models side-by-side.

Provider-agnostic, open-source evaluation infrastructure for language models

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

Comprehensive LLM evaluation at scale: A production-ready framework for evaluating large language models across multiple benchmarks.

CLI port of BenchLocal quality bench packs — runs LLM behavioral evals against any OpenAI-compatible endpoint. Companion to club-3090.

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

discover LLMs punching above their weight

Simple LLM Evaluation Using LLM As a Judge 👩‍⚖️