bentoml/llm-optimizer

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Automating llama.cpp LLM parameter optimization for highest speed/best fit using Morris/Taguchi DoE sweeps.

Local LLM Inference Speed Test Tool

📊 LLM Context Benchmarks - A comprehensive benchmarking tool for testing LLMs with varying context sizes using Ollama. Features dual benchmark modes (API/CLI), automatic hardware detection (optimized for Apple Silicon), visual performance charts.

Optimized GPU compiler for LLM inference. Choose from a list of optimized recipes or optimize your own model via kernel fusion, autotuning, and advanced scheduling. Run benchmarks across different GPU types and configurations, track results and share experiments with the community.

Evidence-driven SGLang inference deployment optimization.

LLM inference decode throughput benchmark with Rich TUI dashboard. Measures token generation speed across concurrency levels and context lengths. Supports SGLang and vLLM engines.

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

Optimizing inference proxy for LLMs