Ion Stoica, Song Han and collaborators from @NVIDIAAIDev, MIT, UC Berkeley and Georgia Tech just released a serving system built for AI agents! ⚡ "Evolving Infra: Agent-Adaptive LLM Serving at Scale"…
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.