Efficiently Serving LLMs

From next-token prediction to serving

The course starts with how a language model chooses its next token, then uses that mechanism to explain why generation is slow. KV caching is the main speed-up it examines. From there the focus moves to writing code that serves an application to many users at once, which is a different problem from running one model call well.

Who benefits

Backend engineers and ML practitioners who have a working model but now face latency or concurrency problems are the natural audience. Understanding the generation loop first makes the serving techniques much easier to reason about.

Key ideas

• Generation repeats a forward pass for every new token, so repeated work is the thing to remove.
• Caching intermediate attention results trades memory for speed.
• Serving many users is a separate engineering problem from making one call fast.