When you have 100K GPUs, you want to utilize all of them.
Given a DL model (like LLM) and thousands of GPUs, how will you distribute your workload efficiently?
Parallelism rules:
- Start with Data Parallelism (DP) = simplest, scales until gradient communication dominates.
- Add Tensor Parallelism (TP) = when layers (matmuls, attention) are too big for a single GPU.
- Add Pipeline Parallelism (PP) = when model depth is too large for memory of one GPU.
- Add Expert Parallelism (EP) = when compute cost grows, use MoE to scale params w/o linear FLOPs.
- Apply ZeRO / FSDP = when optimizer/gradients don’t fit memory.
- Use P/D Disaggregation = when model size exceeds device memory, offload params/states.
- Hybrid Parallelism = real-world = combine DP+TP+PP (+EP/ZeRO) tuned to cluster topology.
Scaling rules:
- Scale outward with DP (more GPUs).
- Scale inward with TP (within GPU ops).
- Scale deep with PP (layers across devices).
- Scale sparse with EP (MoE experts).
