helasaoudi/llm-inspector

Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

A CLI to estimate inference memory requirements for Hugging Face models, written in Python.

Benchmark and optimize LLM inference across frameworks with ease

Live truecolor terminal dashboard for a locally running LLM: tok/s, TTFT, GPUs, context, MTP acceptance, layer and real expert routing (llama.cpp)

GPU Profiler for AI agents

A Lightweight LLM Inference Performance Simulator

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

Extract residual-stream activations and apply steering vectors (including activation oracles) to any vLLM model during inference.