triton-inference-server/server

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

A Datacenter Scale Distributed Inference Serving Framework

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

Universal LLM Deployment Engine with ML Compilation

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

Cross-platform, customizable ML solutions for live and streaming media.

A high-throughput and memory-efficient inference and serving engine for LLMs

Open standard for machine learning interoperability