5 different parallelism techniques:
- Data Parallelism (DP): batch size dimension
- Tensor Parallelism (TP): hidden dimension (number of heads)
- Sequence and Context Parallelism (SP/CP): sequence length dimension
- Pipeline Parallelism (PP): model layers
- Expert Parallelism (EP): model experts/FNN in MoE
Along with DP, use these to reduce memory reduction:
- ZeRO-1: sharding optimizer states among the DP replicas
- ZeRO-2: sharding optimizer states and gradients among the DP replicas
- ZeRO-3: sharding optimizer states, gradients and parameters among the DP replicas
Note:
- ZeRO1/2 can be combined with PP easily. For combining ZeRO3 with PP, additional communication is needed as parameters are split. Global batch size should be large to hide the communication cost. DeepSeekV3 used PP+ZeRO1.
- TP/SP can be combined with PP/ZeRO3 naturally as it is based on distributive property of MatMul and allows weights and activations to be sharded and computed independently.
- TP is model specific. Some ops are sharded on hidden dimension and some on sequence length dimension. Handling this requires model knowledge.
- PP/ZeRO are model agnostic.
- In practice, TP limited to intra-node communication (high speed) and PP/ZeRO-3 for inter-node communication (slow) as it requires less bandwidth.
- For large sequence length (128K+), use CP.
Choose your parallelism technique:
- For small models (< 10B parameters), use TP (TP=8) or ZeRO-3 with DP and Full recompute in 8 GPUs.
- For models with up to 100B parameters, use more than 8 GPUs:
- TP=8 and PP
- Or TP=8 with DP+ZeRO3
- Or only ZeRO3
- If you have up to 512 GPUs, use DP with TP=8 or PP. (ZeRO3 is inefficient).
- If you have more than 1024 GPUs, use TP=8 with ZeRO2 and PP.
- If you use very large sequence length, use CP in addition.
- If your model is MoE, use EP in addition.
- If you have few GPUs (poor researcher), try full activation re-computation, increase gradient accumulation to process larger batch size.
- If still does not work, sorry. Get a job, save some money and buy more GPUs. If this is for your job, ask your manager and enjoy free time till they met your demand.
