NVIDIA/TensorRT-Edge-LLM

LLM inference in C/C++

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

LiteRT, successor to TensorFlow Lite. is Google's On-device framework for high-performance ML & GenAI deployment on edge platforms, via efficient conversion, runtime, and optimization

Efficient AI Inference & Serving

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

LLM inference in C/C++

lightweight, standalone C++ inference engine for Google's Gemma models.