Prasannajaga/inference-book

Day 4/30 of inference infrastructure series Vllm & sglang today we look at how actual LLM serving engines execute requests in production focusing on what happens inside a single request in a serving…

Day 3/30 of inference infrastructure series Kv cache & Quantization Part 1: kv cache: the state behind LLM inference last day on day 2 we broke down how prefill and decode run on the gpu prefill…

Part 2: Decode In decode, the GPU generates one token at a time autoregressively in a loop, at each step, the model takes your newly generated token and runs another forward pass to predict the next…

Day 1/30 of inference infrastructure series Part 1: Request Lifecycle The request lifecycle shows you how your query goes through the server infrastructure the big difference today is theres no…

I dont know whether this is the new algorithm update or just the ragebait title, but this post blew up and became my highest engagement post yet! A small intro about me, Im an inference engineer…