vllm.cpp mirrors vLLM by default, but some of the good scheduling ideas I always fancied are sglang's, so vllm.cpp supports different set of features: --enable-radix-attention (fused into the…

been writing a C++20 port of vLLM for a while now, and was quite silent on it, because I don't like to talk until I have results. So I was in "cave mode". Name TBD. Until then it's vllm.cpp paged KV…

LocalAI unveils vllm.cpp, a 66 MB C++ inference server It contains - continuous batching - tool calling - structured output - an OpenAI-compatible API So basically we truly have AI in our pockets now