huggingface/optimum-benchmark

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

🤗 Optimum Intel: Accelerate inference with Intel optimization tools

Measure and optimize the energy consumption of your AI applications!

Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.

Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

A CLI to estimate inference memory requirements for Hugging Face models, written in Python.

TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs