vllm-project/vllm-neuron

Training and inference on AWS Trainium and Inferentia chips.

A native PyTorch backend for AWS Trainium with support for eager execution, torch.compile, distributed training, and custom NKI kernels

Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and integrated with your favorite AWS services

A high-throughput and memory-efficient inference and serving engine for LLMs

The Runpod worker template for serving our large language model endpoints. Powered by vLLM.

vLLM plugin for RBLN NPU

Nano vLLM