happyme531/open_rknpu2_lm_infer

Llama.cpp with the Rockchip NPU integration as a GGML backend.

Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )

Run RKLLM only in one docker command. Already converted models here.

Drop-in ggml backend for Rockchip NPUs: offloads llama.cpp/whisper.cpp prefill to the RK3588 NPU

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

LLM inference in C/C++

NPU Ollama - An Ollama/OpenAI compatible API for Intel OpenVINO compatible computers

High-performance single-GPU inference for selected model checkpoints and GPUs.