SemiAnalysisAI/InferenceX

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Reference implementations of MLPerf® inference benchmarks

Multiple NVIDIA GPUs or Apple Silicon for Large Language Model Inference?

TokenSpeed is a speed-of-light LLM inference engine.

Cost-efficient and pluggable Infrastructure components for GenAI inference

A high-throughput and memory-efficient inference and serving engine for LLMs

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

100 days of LLM inference engineering — daily posts, experiments, and visualizations