Skip to content

Hybrid Parallelism and Placement#

concept

Large jobs combine several strategies. The total number of GPUs is the product of the parallel degrees:

world_size = TP × CP × PP × DP
# EP is carved out of the DP × CP ranks for the MoE layers

Choosing the degrees is half the problem. The other half is placement: deciding which physical GPU gets which rank, so each dimension's traffic lands on a link fast enough for it.

Rank ordering#

Frameworks assign ranks so that the innermost dimension varies fastest. Megatron-Core's default order is tp-cp-ep-dp-pp: consecutive ranks form a TP group, then CP, then EP, DP, and PP outermost. If the launcher maps consecutive ranks to the GPUs of one node, TP automatically lands on NVLink.

Rank order only works if placement cooperates

The framework assumes ranks 0–7 share a node. If the scheduler or launcher scatters them, TP traffic silently crosses the fabric. Nothing fails; the job is just several times slower.

Worked example: TP=8, PP=4, DP=8 on 256 GPUs#

32 HGX nodes, 8 GPUs each, with one 400G NIC per GPU on a rail-optimized fabric (GPU i of every node connects to leaf switch i).

              each node = one TP group of 8 GPUs (NVLink)
              DP replica 0   DP replica 1   …   DP replica 7
PP stage 1    node 0         node 1         …   node 7
PP stage 2    node 8         node 9         …   node 15
PP stage 3    node 16        node 17        …   node 23
PP stage 4    node 24        node 25        …   node 31

one pipeline (= one DP replica): node 0 → 8 → 16 → 24
Dimension Group Traffic Link it uses
TP = 8 the 8 GPUs of one node 4 all-reduces per layer, blocking NVLink / NVSwitch
DP = 8 GPU i on the 8 nodes of one PP stage gradient all-reduce, overlapped Rail i: one leaf switch, no spine hop
PP = 4 GPU i on node n, n+8, n+16, n+24 point-to-point activations Rail i, crossing to the next stage's nodes

Because each GPU's DP and PP peers sit on the same rail, inter-node traffic mostly stays within one leaf switch. That is the reason for rail-optimized fabrics.

How Kubernetes makes this placement possible#

Placement is not automatic. On Kubernetes it is the result of several components agreeing:

Need Mechanism
All 256 pods start together or not at all Gang scheduling: Kueue admission, Volcano PodGroup, JobSet
One pod per node owning all 8 GPUs Pod requests nvidia.com/gpu: 8, or a DRA claim for the whole node
Pods packed into the same rack / leaf / NVLink domain Kueue topology-aware scheduling, node labels for rack and block, DRA compute domains for multi-node NVLink
GPU and NIC on the same PCIe root / NUMA node Kubelet Topology Manager, SR-IOV/RDMA device plugins with NUMA info
Rank r on the right host Launcher (torchrun, MPI, JobSet) maps pod index to node rank deterministically

Choosing the degrees: a practical order#

  1. TP: the smallest value that makes one layer fit, at most the NVLink domain size (usually 8).
  2. CP: only if activations still do not fit at the target sequence length.
  3. PP: only if the model still does not fit in one TP×CP group; prefer the smallest value, and use enough micro-batches to keep the bubble small.
  4. EP: for MoE models, sized so experts spread evenly and most all-to-all stays inside NVLink.
  5. DP: everything that is left, to reach the target throughput. Use FSDP/ZeRO inside DP to trade communication for memory if needed.

Then validate: run nccl-tests on the exact placement, profile one step, and check that GPU idle time lines up with the expected communication phases. The co-design page covers how to read those signals.