naklecha/simple-llm

Based on Nano-vLLM, a simple replication of vLLM with self-contained paged attention and flash attention implementation

Runs 405B LLMs on 8GB VRAM

Nano vLLM

High-speed Large Language Model Serving for Local Deployment

Pure Rust Inference Engine

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

A high-throughput and memory-efficient inference and serving engine for LLMs

Experimental llama.cpp fork for inference research and development

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.