microsoft/onnxruntime-genai

Run Generative AI models with simple C++/Python API and using OpenVINO Runtime

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

A fast inference library for running LLMs locally on modern consumer-class GPUs

High-speed Large Language Model Serving for Local Deployment

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

Efficient AI Inference & Serving

A high-throughput and memory-efficient inference and serving engine for LLMs

Run frontier LLMs and VLMs locally on Qualcomm devices across NPU, GPU, and CPU with a few lines of code

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.