airockchip/rknn-llm

LLM inference in C/C++

llama.cpp fork with additional SOTA quants and improved performance

Fast, flexible LLM inference

Run LLMs on AMD Ryzen™ AI NPUs in minutes; purpose-built and deeply optimized for the AMD NPUs.

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

LLM inference in C/C++

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.