pytorch/serve

A flexible, high-performance serving system for machine learning models

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Containers for machine learning

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

Open standard for machine learning interoperability

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.