AI course essential

5 different parallelism techniques

LLMs are memory hungry. GPUs are limited by memory more compared to compute.

[CHEATSHEET] Separate GPUs for Prefill and Decode.

[CHEATSHEET] Use multiple GPUs and pipeline.

[CHEATSHEET] This works best with continuous batching + chunk prefill.

[CHEATSHEET] Optimization that makes server performance realistic.

[CHEATSHEET] Understand the overhead of LLM Inference.

Inference = Prefill + Decode

In safety-critical, long-lifecycle systems = obsolescence is the real bottleneck. Adaptive architectures (FPGAs, SoCs, stable CPUs) = keep innovation alive after deployment.

When you have 100K GPUs, you want to utilize all of them.

Understanding how to do calculations in INT8 instead of FP32 and still get same results is a core optimization in DL. DL is highly insensitive to local errors.

The embedding layer (before Attention layer) expands each token to a 1D vector of size hidden_dim. The common sizes of hidden dimension are 1024 to 8096.

LLM is memory intensive. This limits the LLM that can run on given GPUs. Calculating maximum context length supported in a given hardware + model

The country that will control AGI will control the future of humanity.