Debugging Slow Collectives

concept

A decision tree for the most common complaint on a GPU cluster: training runs, but much slower than it should. It works from the inside out: first prove one node is healthy, then the RDMA path, then the fabric, then placement. Each check names the layer where the fix lives.

Start by measuring. Run all_reduce_perf from nccl-tests and compare its bus bandwidth (busbw) with what the hardware should deliver. For example, a 400G NIC carries about 50 GB/s per direction, and a healthy inter-node all-reduce reaches a large fraction of that per GPU.

The decision tree

flowchart TD
    A([busbw well below expected]) --> B{Slow on a<br/>single node too?}
    B -- Yes --> C[L8: check NVLink and P2P<br/>nvidia-smi topo -m<br/>nvidia-smi nvlink -s]
    B -- No, only multi-node --> D{NCCL_DEBUG=INFO:<br/>NET/IB or NET/Socket?}
    D -- NET/Socket --> E[L6 + control plane:<br/>RDMA device not in pod,<br/>IPC_LOCK missing,<br/>NCCL_IB_HCA wrong]
    D -- NET/IB --> F{Log shows<br/>GDRDMA?}
    F -- No --> G[L6/L8: GPUDirect off<br/>nvidia-peermem<br/>or dma-buf,<br/>ACS / IOMMU settings]
    F -- Yes --> H{Each GPU paired<br/>with its own NIC?<br/>PIX or PXB in topo}
    H -- No, SYS or NODE --> I[L8 + control plane:<br/>wrong NIC mapping,<br/>Topology Manager policy]
    H -- Yes --> J{One node or link<br/>slower than the rest?}
    J -- Yes --> K[L7: bad link or port<br/>ib_write_bw between pairs,<br/>port error counters]
    J -- No --> L{RoCE: pause frames<br/>or ECN marks climbing?}
    L -- Yes --> M[L7: congestion config<br/>PFC, ECN, DCQCN tuning]
    L -- No --> N[Control plane + L3:<br/>rank placement,<br/>TP crossing nodes,<br/>rails crossing spine]

The checks, with commands

#	Check	How	Healthy result
1	Single-node bandwidth	all_reduce_perf -b 8 -e 8G -f 2 -g 8 on one node	Close to NVLink bandwidth for the GPU generation
2	NVLink topology	nvidia-smi topo -m	NV# between every pair of GPUs
3	NVLink errors	nvidia-smi nvlink -s and -e	All links up, error counters not growing
4	RDMA in use	NCCL_DEBUG=INFO, look for NET/IB : Using [0]mlx5_...	NET/IB, never NET/Socket for GPU traffic
5	GPUDirect RDMA	Same log, look for via NET/IB/<n>/GDRDMA	GDRDMA on inter-node channels
6	GPU-NIC affinity	nvidia-smi topo -m, GPU rows vs NIC columns	Each GPU has PIX or PXB to its NIC, not SYS
7	Point-to-point links	ib_write_bw between each pair of nodes	Every pair close to line rate; no outlier
8	Port errors	perfquery (InfiniBand) or ethtool -S (RoCE)	Symbol, link-down and discard counters flat
9	Congestion (RoCE)	ethtool -S: pause and ECN/CNP counters	Low and stable under load, no pause storms
10	Placement	Rank-to-node map against the fabric topology	TP inside a node; DP groups on the same rail

Why the order matters

The checks run from the inside out because each one assumes the ones before it passed. There is no point tuning ECN (check 9) while NCCL is running over TCP (check 4). The two silent failures, TCP fallback and GPUDirect off, are also the most common and the cheapest to find: both show up in the first lines of an NCCL_DEBUG=INFO log.

See from pod spec to data path for how the RDMA device and IPC_LOCK reach the pod, and life of a training step for where each hop sits in a step.