larq/compute-engine

High-efficiency floating-point neural network inference operators for mobile, server, and Web

ncnn is a high-performance neural network inference framework optimized for the mobile platform

CMSIS-NN Library

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

Neural Network Compression Framework for enhanced OpenVINO™ inference

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

Humming is a high-performance, lightweight, and highly flexible JIT (Just-In-Time) compiled GEMM kernel library specifically designed for quantized inference.