Decode-Maximal Batching

[CHEATSHEET] This works best with continuous batching + chunk prefill.

Problem: Decode is sequential and under-utilizes GPUs (<20% FLOPs).

Solution: Always maximize the batch size of active decode requests per step, so the GPU runs one large kernel instead of many tiny ones.

At decode step t:

• Suppose active requests = 50.
• Instead of launching 50 tiny decodes:
• Inputs merged → [B=50, 1, d_model]
• Outputs → [B=50, 1, V]

Each request still attends its own KV cache:

• KV_i (KV cache of i^th decode): [1, n_heads, (L_i + t), d_head]

Gains

• Higher arithmetic intensity → GPU kernels more efficient.
• Less launch overhead → fewer kernel invocations.
• Fair scheduling → all active requests progress together.

Powerful when combined with Continuous Batching → new requests can join mid-flight.