Skip to content

Parallelism#

concept

Parallelism is how one training or inference job is spread across many GPUs. Every strategy splits something different, and each split creates a different kind of network traffic. That traffic is what the layers below (NCCL, transport, fabric, interconnect) have to carry, so the parallelism choice is where infrastructure design starts.

Why parallelize at all#

There are only two reasons:

  1. The model does not fit. Weights, gradients, optimizer state and activations exceed one GPU's memory.
  2. The job takes too long. Even if it fits, one GPU would need months to process the training data, or cannot serve the request rate.

Reason 1 forces you to split the model. Reason 2 lets you replicate the model and split the data. Real jobs usually face both.

Memory math: why big models do not fit#

Training with mixed precision and the Adam optimizer needs roughly 16 bytes per parameter before counting activations:

Item Precision Bytes / parameter
Weights BF16 2
Gradients BF16 2
Master weights FP32 4
Adam momentum FP32 4
Adam variance FP32 4
Total 16

For a 70B-parameter model that is 70e9 × 16 B ≈ 1.12 TB of state, about 14× the memory of one 80 GB H100, and activations come on top. Fine-tuning with frozen weights (LoRA) or serving (weights only, plus KV cache) needs far less; see HBM and memory planning.

The strategies at a glance#

Strategy What is split Main collective When it runs Sensitivity Typical placement
Data parallel (DP) The batch All-reduce of gradients Once per step, overlapped Bandwidth Across nodes
FSDP / ZeRO Batch + model state All-gather, reduce-scatter Every layer, prefetched Bandwidth Within or across nodes
Tensor parallel (TP) Each layer's matrices All-reduce (or RS + AG) Every layer, on the critical path Latency and bandwidth Inside one NVLink domain
Pipeline parallel (PP) Groups of layers Point-to-point send/recv Between stages, per micro-batch Latency Across nodes
Context parallel (CP) The sequence Ring exchange of K/V Every attention layer Bandwidth Inside or near the NVLink domain
Expert parallel (EP) MoE experts All-to-all Every MoE layer, twice Bandwidth, congestion, tail latency Inside the NVLink domain, then across a rail

The placement rule#

The chattiest dimension goes innermost, on the fastest link.

fastest link ─────────────────────────────────────────► slowest link
NVLink/NVSwitch                                  RDMA fabric
(H100: 900 GB/s per GPU, bidirectional)          (400G NIC: 50 GB/s per direction)

TP            ──►  CP / EP        ──►  DP / FSDP       ──►  PP
every layer,       every layer,        once per step,       point-to-point,
blocking           large volumes       overlappable         small, latency-bound

TP inside a node, CP and EP next, DP across nodes, and PP across the longest distances. Hybrid parallelism and placement shows how the dimensions combine and how they map onto real cluster topology.

Reading order#