viitrix/vt-transformer

~1300 line, minimal, extensible LLM inference engine built from scratch.

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

Efficient AI Inference & Serving

Optimized GPU compiler for LLM inference. Choose from a list of optimized recipes or optimize your own model via kernel fusion, autotuning, and advanced scheduling. Run benchmarks across different GPU types and configurations, track results and share experiments with the community.

Pure Rust Inference Engine

Based on Nano-vLLM, a simple replication of vLLM with self-contained paged attention and flash attention implementation

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡