triton-inference-server/model_analyzer

Offline optimization of your disaggregated Dynamo graph

A library to analyze PyTorch traces.

A profiling and performance analysis tool for machine learning

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

A CLI to estimate inference memory requirements for Hugging Face models, written in Python.

Serve, optimize and scale PyTorch models in production

Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

🤗 Optimum Intel: Accelerate inference with Intel optimization tools