OpenNMT/CTranslate2

Transformer related optimization, including BERT, GPT

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

lightweight, standalone C++ inference engine for Google's Gemma models.

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

Hackable and optimized Transformers building blocks, supporting a composable construction.

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Hummingbird compiles trained ML models into tensor computation for faster inference.