Curriculum

Week 14. Scaling laws, precision, parallelism

Phase 3. Inference, GPUs, scaling · week 14 of 24

Learn in the app: tutor, coding problems →

Core: Chinchilla, collectives, DDP, ZeRO and FSDP, tensor and pipeline parallelism, bf16 versus fp16, absmax quantization · Depth: FP8 and accumulation precision, parallelism for inference, MoE problems · ≈ 14 h core / 25 h total

Week 9 tells you what a model costs; this week is about how to spend the budget and the hardware. Scaling laws decide which model to train and on how many tokens; number formats decide what to pack it into; parallelism decides how to spread it across GPUs when 16 bytes per parameter do not fit on one. Without this you cannot answer either "how much does training cost" or "5D parallelism".

In plain terms. The training budget is C ≈ 6·N·D FLOPs (week 9), where N is the parameters and D is the tokens. Chinchilla tells you how to split this budget: about 20 tokens per parameter. A 1B-parameter model needs 20B tokens: 6 · 10⁹ · 2·10¹⁰ = 1.2·10²⁰ FLOPs, which on an H100 at 40% MFU is about 84 GPU-hours. A model twice as large on the same budget sees half as many tokens, and its loss ends up worse than the optimal one's. Llama 3 8B was trained on ~15T tokens: about 1900 per parameter, almost 100 times the "optimum". Training costs more, but a small model is cheaper to serve every day.

• Kaplan vs Chinchilla: compute-optimal (the best loss for a fixed FLOPs budget), a token-to-parameter ratio of ~20:1, and why in practice people train much longer (inference cost)
• Loss prediction, emergence (the abrupt appearance of a capability as scale grows), and critiques of the concept itself: the jump is often created by an all-or-nothing metric, not by the model

In plain terms. In fp16 the number 70,000 no longer fits and becomes inf, and a gradient of 10⁻⁸ turns into 0. bf16 holds both: its range is the same as fp32, up to ~3·10³⁸. But bf16 has only 8 significant bits, and 257 rounds to 256: representable neighbors there are 2 apart. fp16 stores 257 exactly. Exponent bits give range, mantissa bits give precision, and with 16 bits you have to choose. That is why fp16 needs loss scaling (multiplying the loss so small gradients do not vanish), and bf16 usually does not.

• Precision: fp32, tf32, bf16, fp16, fp8. Range vs mantissa (the significant bits of a number)
• Quantization (storing weights, activations or the KV cache in fewer bits): int8, int4, GPTQ, AWQ, QAT (quantization-aware training). This is the course's main reference point: weeks 11, 13 and 15 point here

In plain terms. Quantize the vector (0.3, −0.5, 0.12, 1.5) to int8 with absmax (scaling by the maximum absolute value): grid step scale = 1.5 / 127 ≈ 0.0118, q = round(x / scale) = (25, −42, 10, 127), and back q·scale = (0.295, −0.496, 0.118, 1.5). Now replace 1.5 with an outlier of 30. The step grows to 0.236 and you get (1, −2, 1, 127): 0.3 and 0.12 have merged, and all numbers up to 0.5 in absolute value got 5 levels (from −2 to 2) out of 255. Cut the vector into blocks of 2, each with its own step, and the first pair gets (76, −127): the outlier spoiled only its own block.

• Absmax: scale = max|x| / (2^(b−1) − 1) (the grid step), q = round(x / scale), x ≈ q·scale. The scale can be per tensor, per channel or per block. The smaller the block, the less an outlier hurts, but the more scales you store. MX formats (microscaling) give every 32 numbers a shared scale; for FP8 training DeepSeek-V3 used its own scale for every 128 activation values and for each 128×128 block of weights. The scale is computed from the current block (online), not from the maxima of previous steps (delayed scaling): that way an outlier does not catch you off guard
• FP8 comes in two kinds: E4M3 (4 exponent bits, 3 mantissa, max 448) is more precise, E5M2 (5 and 2, max 57,344) has more range. The classic recipe: E4M3 for weights and activations, E5M2 for gradients. With fine-grained block scaling, E4M3 is enough everywhere. Matmuls run in FP8, while embeddings, the output head, norms, master weights and optimizer states stay in bf16 or fp32
• Accumulation precision. In plain terms. In E4M3, between 64 and 128 neighboring numbers are 8 apart, so 64 + 3 = 64. Add up 100 threes, rounding the sum to E4M3 after each step: after 18 steps you reach 64 and get stuck, ending at 64 instead of 300. That is why FP8 products are summed in a wider accumulator, with partial sums regularly promoted to fp32: promoting every 8 terms, the same 100 threes give exactly 300. DeepSeek-V3 noticed that the tensor core accumulator in FP8 keeps only about 14 bits, and promoted sums to fp32 at fixed intervals

Code → nanolm/quant.py: quantize_symmetric (per tensor and per channel), quantize_affine with a zero point (the code that 0.0 maps to exactly), quantize_blockwise, QuantizedLinear (int8 and a scale per output channel), quantize_model and model_size_bytes. The task is in exercises_en/quant.py, check: NANOLM_IMPL=exercises_en pytest tests/test_quant.py -v. In the code scale is the same grid step as in the example above (x ≈ q·scale), which is also how PyTorch defines it. Check by hand: an outlier in one row hurts per-tensor quantization more than per-channel and per-block, and NanoLM after quantize_model barely changes its loss, although the layer weights became almost 4 times lighter.

In plain terms. 4 GPUs, each with its own 4-number gradient: (1, 2, 3, 4), (5, 6, 7, 8), (9, 10, 11, 12), (13, 14, 15, 16). Everyone needs to end up with the sum (28, 32, 36, 40). Reduce-scatter: GPU 0 collects only the first elements and gets 1 + 5 + 9 + 13 = 28, GPU 1 collects the second ones and gets 32, and so on. Each has sent three of its numbers and holds one sum. All-gather: each sends its sum to the other three: three more numbers. In total, 6 numbers sent per GPU for an array of 4. In general, 2·(p−1)/p of the array size for p GPUs: with 8 GPUs that is 1.75×, with 1000 almost 2×, so the cost barely grows with the number of GPUs.

• Collective operations (exchanges in which all GPUs of a group take part): broadcast, reduce, all-reduce, all-gather, reduce-scatter (all-reduce = reduce-scatter + all-gather)

In plain terms. Four GPUs are four people, each with the same notebook: weights, gradients, optimizer notes. In DDP (plain data parallelism) everyone has a full copy. For 7B with mixed precision that is 16 bytes per parameter (week 9), 112 GB on every GPU before activations; it does not fit in 80 GB. ZeRO splits the notebook four ways. ZeRO-1 splits only the optimizer notes (12 bytes out of 16): 2 + 2 + 12/4 = 7 bytes per parameter, 49 GB. ZeRO-2 also splits the gradients: 2 + 2/4 + 3 = 5.5 bytes, 38.5 GB. ZeRO-3 splits the weights themselves too: 16 / 4 = 4 bytes, 28 GB. A layer's weights are then collected with all-gather right before use and discarded immediately after.

• Kinds of parallelism:
  • Data parallelism (each GPU holds the whole model and processes its share of the batch), ZeRO stages 1/2/3, FSDP (PyTorch's implementation of ZeRO-3)
  • Tensor parallelism (one layer matrix is split across GPUs; column/row splitting, and where an all-reduce is needed)
  • Pipeline parallelism (layers are spread across GPUs, microbatches flow through a pipeline), the pipeline bubble (the fraction of time some GPUs wait: with 4 stages and 8 microbatches it is 3/11 ≈ 27%), 1F1B (a schedule in which microbatch forwards and backwards alternate)
  • Sequence / context parallelism (the sequence length is split), expert parallelism (MoE experts on different GPUs)
  • When tensor parallelism hits the communication limit. Split the matmul (B×D)·(D×F) along the inner dimension D across 2 GPUs. Each computes for B·D·F / P seconds and then contributes B·F·2 bytes to an all-reduce, taking 2·B·F / W_net seconds. B and F cancel: compute takes longer than communication when D > 2·P / W_net. An H100 with NVLink (about 450 GB/s one way) gives D > 4,400: for Llama-3-8B (D = 4096) communication is already on par with compute, and the inter-node network is several times slower than NVLink. That is why TP is kept within a node. For inference, the same reasoning continues below, in "Parallelism for inference"
  • Hence "5D parallelism", a question interviewers ask word for word

• Choosing a strategy: what is memory-bound, what is communication-bound

Parallelism for inference. Everything above was about training. When serving a model the goal is different: not to fit the optimizer state but to meet TTFT and TPOT (week 11). The decode step numbers come from step 4 of week 11.

In plain terms. A 2,000-token prompt costs about 3.1·10¹³ FLOPs: at 50% MFU on an H100 that is 63 ms. Insert such a prefill between the decode steps of a batch of 32 (a 7.6 ms step at a context of about 2,250), and all 32 users stall for more than 8 steps: their tokens arrive in bursts. Hand the prefill to a separate machine and the bursts go away, but the prompt's cache (262 MB) now has to be sent over the network.

• Sharding for inference. Prefill looks like training without backward: tensor parallelism works, and for very long prompts so does splitting along the sequence. In decode the choice is narrower. FSDP hurts: gathering weights over NVLink (about 450 GB/s one way) is 7 times slower than reading them from your own HBM. Data parallelism does not speed up a step: it is just independent replicas. There is no sequence to split, the new token is a single one. What remains is tensor parallelism: each GPU reads its share of the weights and its share of the cache (the cache is split by KV heads), and only activations go over the network
• What this costs, by example: Llama-3-8B, B = 32, 8k. On 2 GPUs the step goes 15.1 → 7.5 ms. But each step has 64 all-reduces (two per layer) of B·D·2 = 256 KiB. Bandwidth-wise that is a fraction of a microsecond, but each one has a launch latency on the order of 10 µs, about 0.6 ms per step. On 8 GPUs the step is 1.9 ms, and communication adds another third to it. You shard for latency or when the model does not fit on one GPU; for throughput, independent replicas are cheaper. If there are more GPUs than KV heads, the cache is additionally split by batch
• How to combine prefill and decode. (1) A shared batch: simple, but every prefill stalls decode, as in the example. (2) Chunked prefill (Sarathi-Serve): the prompt is cut into chunks that are mixed into decode steps. A step at B = 32 is underloaded on arithmetic: 32 decode tokens plus 256 prompt tokens give 2·N·288/P = 4.4 ms of compute, still less than the 4.79 ms of weight reading, so the prompt chunk rides along almost for free. (3) Disaggregated serving (separate pools; DistServe, Splitwise): some GPUs do only prefill, others only decode, and the cache is sent over the network. The pools are scaled independently: prefill is responsible for TTFT, decode for TPOT
• Sizing the pools, by example: a 2,000-token prompt, a 500-token response. A prefill GPU handles 1 / 0.063 ≈ 16 requests/s. A decode GPU at B = 32 takes a 7.6 ms step and 500 steps per batch: 32 / (500 · 7.6 ms) ≈ 8.4 requests/s. You need about two decode GPUs per prefill GPU. The longer the responses, the larger the decode share

Practice: draw up a concrete plan for training a 7B model on 8×H100 and on 512×H100: which parallelism, what batch size, estimated time and cost. Defend your choice out loud.

Back-of-the-envelope problems. A node has 8 H100s, NVLink gives 450 GB/s per GPU, the inter-node network 50 GB/s per GPU, MFU is 40%. A collective over p participants pushes about (p − 1)/p of the array through each GPU (all-reduce twice that).

1. FSDP for 7B within one node, 16,384 tokens per GPU per step. How much time goes to communication and how much to compute?
2. The same FSDP is stretched over 64 nodes (512 GPUs). What happened to communication, and how do you fix it?
3. 70B on 64 H100s (8 nodes). Does the training state fit with FSDP only within a node? Across all 64 GPUs? What is a third option?
4. MoE: 24 layers, D = 2048, a SwiGLU expert with 3·D·F parameters at F = 1024, 64 routed and 2 shared experts, top-6, attention 4·D² per layer, embeddings not counted. How many parameters in total and how many are active per token? How much do the weights take per GPU with expert parallelism over 8 GPUs?
5. The same MoE, 8,192 tokens per GPU. How many bytes go into all-to-all per layer in the forward pass, and how long does that take within a node and across nodes? Compare with the layer's compute at 50% MFU.

Answers. (1) The bf16 weights take 14 GB. Each step has three collectives: an all-gather of weights in forward, another in backward and a reduce-scatter of gradients. Each is 7/8 · 14 ≈ 12.3 GB, 27 ms; about 80 ms in total. Compute is 6 · 7·10⁹ · 16,384 / (989·10¹² · 0.4) ≈ 1.7 s. Communication is about 5% and gets hidden behind compute. (2) Every collective now goes over the inter-node network: 14 / 50 ≈ 0.28 s, three collectives 0.84 s, half of compute. The fix is HSDP (hybrid sharding): the state is sharded within a node and replicated across nodes. Then only an all-reduce of each GPU's eighth of the gradients crosses nodes: 2 · 1.75 / 50 ≈ 0.07 s. (3) The state is 16 · 70·10⁹ = 1.12 TB. On 8 GPUs that is 140 GB each: it doesn't fit. On 64 GPUs it's 17.5 GB, with about 60 GB left for activations. The third option: TP = 8 within a node and ZeRO-1 across nodes, 8.75·10⁹ · (2 + 2 + 12/8) ≈ 48 GB. (4) An expert is 3 · 2048 · 1024 ≈ 6.3M, attention 16.8M. A layer is (64 + 2) · 6.3 + 16.8 ≈ 432M, about 10.4B in total. Active: (6 + 2) · 6.3 + 16.8 ≈ 67M per layer, 1.6B in total: such an MoE computes like a 1.6B model but takes memory like a 10.4B one. With EP = 8, each GPU holds 8 routed experts of every layer plus a copy of the shared experts and attention: (8 + 2) · 6.3 + 16.8 ≈ 80M per layer, 1.9B parameters, 3.8 GB. (5) Dispatching tokens: 8,192 · 6 · 2,048 · 2 ≈ 201 MB, the same again to combine; 7/8 goes to other GPUs, about 350 MB. Within a node 0.35 / 450 ≈ 0.8 ms, across nodes 7 ms. The layer's compute is 2 · 67·10⁶ · 8,192 / (989·10¹² · 0.5) ≈ 2.2 ms. Within a node, communication hides behind compute; across nodes it takes three times longer than compute. That's why expert parallelism is kept within a node, or the number of nodes a single token can be sent to is capped.

Math (Track D): D19: how many tokens you need to estimate the loss; D20: compute-optimal model size via Lagrange multipliers.

Interview question of the week: "How do you train a 7B model on 8×H100? Which parallelism, and why?" A 3-minute structure: (1) memory comes first, not FLOPs: 16 bytes per parameter → 112 GB on each GPU with DDP, which does not fit in 80 GB even before activations; (2) conclusion: shard the state with ZeRO/FSDP; ZeRO-3 on 8 GPUs leaves 16/8 = 2 bytes per parameter, 14 GB; ZeRO-1 adds no communication over DDP: an all-reduce already is reduce-scatter + all-gather; (3) activations are counted separately: for Llama-3-8B at batch 8 × 8192 they are 384 GiB (budget.py), cut down by checkpointing and microbatches; (4) communication: an all-reduce costs 2·(p−1)/p of the array, 1.75× on 8 GPUs; pipeline only if the layers do not fit, with a bubble of 3/11 ≈ 27% at 4 stages and 8 microbatches; (5) time: 6N × tokens / (8 × 989 TFLOPs × MFU), with tokens at ~20 per parameter per Chinchilla, or more for cheaper inference; (6) expect "and on 512×H100?": most of the scaling comes from data parallelism, and the limits are the global batch and inter-node communication.

Sources: Hoffmann et al., Chinchilla (2022); Micikevicius et al., Mixed Precision Training (2018) and FP8 Formats for Deep Learning (2022); Rouhani et al., Microscaling Data Formats (2023); DeepSeek-V3 (2024), FP8 training with block scaling and promotion of sums to fp32; Rajbhandari et al., ZeRO (2020); for inference Agrawal et al., Sarathi-Serve (2024); Zhong et al., DistServe (2024); Patel et al., Splitwise (2024).

Deeper: 05-ГЛУБИНА, sections "★★ Week 14. Parallelism: compute it, do not just name it", "Week 14. μP and fitting scaling laws" and "Week 14. Precision: practice, not just theory".

Week outcomes

• I can draw up a parallelism plan for 7B on 8×H100 and on 512×H100 and defend it out loud.
• I can lay out the collectives for a full transformer layer at TP = 8.
• I can explain why ZeRO-1 adds no communication compared to DDP.
• I can compute the compute-optimal model size and token count for a given budget using Chinchilla.
• I can quantize a vector with absmax by hand and explain why FP8 needs block scaling and fp32 accumulation.
• I can explain why decode is sharded only with tensor parallelism, and when prefill is moved to separate GPUs.

Self-check

1. Which two collectives make up an all-reduce, and why does an all-gather in the forward pass correspond to a reduce-scatter in the backward?
2. bf16 versus fp16: how do their range and mantissa differ, and when do you choose which?
3. Where does the pipeline bubble come from, and what does its size depend on?
4. Why does a single outlier ruin int8 quantization of a whole tensor, and how does block scaling fix it?

✅ Checkpoint 3

Rapid-fire, 30 questions in 30 minutes, out loud, covering all of Phase 3.