Skip to content

Hardware/Software Co-Design#

concept

A distributed AI job is a tightly coupled system. Its speed is set by its slowest component on its slowest rank, so tuning one layer in isolation rarely moves the result. Co-design means choosing and validating the parallelism, placement, communication, fabric, GPU topology and storage together, starting from the workload.

Why the slowest rank sets the pace#

Every collective is a synchronization point. An all-reduce completes only when the last participant contributes. If one of 256 GPUs is 10% slower (a hot GPU throttling, a NIC on the wrong NUMA node, a congested link), all 256 wait for it at every collective.

step_time ≈ max over ranks r ( compute[r] + exposed_comm[r] + data_wait[r] )

Averages hide this. Per-rank, per-step timing is what reveals it.

Worked example: one job, every layer#

A dense transformer trained with TP=8, PP=4, DP=8 on 256 H100 GPUs.

Layer Decision Consequence if wrong
L1 Application Global batch size and sequence length Too few micro-batches per pipeline: large PP bubble
L2 Framework Megatron-Core with SP, activation recomputation, overlapped DP all-reduce Out-of-memory, or communication left exposed
L3 Parallelism TP inside the node, PP across 4 nodes, DP across 8 Wrong order puts TP on the fabric: several-fold slowdown
L4 Runtime and kernels Fused attention kernels, BF16/FP8 math libraries, CUDA version matched to the driver GPUs busy but slow: low tensor-core use per step
L5 NCCL IB transport, GPUDirect RDMA, correct HCA list Silent fallback to TCP sockets: 10× slower collectives
L6 Transport Matching user-space verbs/libfabric in image and kernel modules on host RDMA device missing in pod, or errors under load
L7 Fabric Rail-optimized, non-blocking leaf-spine; tuned ECN/DCQCN or adaptive routing Congestion hotspots; PFC pause storms stall unrelated jobs
L8 Interconnect GPU↔NIC on the same PCIe switch; full NVSwitch connectivity Cross-socket traffic cuts NIC bandwidth
L9 GPU HBM sized for TP-sharded weights plus activations; clocks not throttling OOM, or one throttled GPU slowing all 256
Control plane Gang admission, whole-node pods, rack/leaf topology awareness Partial starts waste GPU-hours; scattered pods cross spines
Storage Data from a parallel FS with local NVMe cache; async checkpointing GPUs idle waiting for data; long checkpoint stalls

A bottleneck taxonomy#

When throughput is lower than expected, the time went to one of six places:

Bottleneck Symptom First signals to check
Compute High SM activity, step time matches FLOP estimate DCGM SM active/occupancy, tensor-core activity
Memory OOM, or small batches forced by memory HBM used, allocator fragmentation
Intra-node communication Idle gaps inside each layer NVLink throughput, NCCL kernel time for TP collectives
Inter-node communication Idle gaps at DP/PP/EP boundaries NCCL time, NIC throughput, RDMA retransmits, PFC pauses, ECN marks
Input / storage Idle at step start; spiky step times Data-loader wait time, file-system read throughput, checkpoint duration
Scheduling / placement Some ranks consistently slow; jobs waiting Pod-to-node map, topology labels, queue wait time

Correlating across layers#

The diagnostic method is to line up signals from different layers on one timeline:

flowchart LR
  G["GPU idle (DCGM)"] --> N["NCCL collective time (profiler / NCCL logs)"]
  N --> F["Fabric counters (RDMA retransmits, PFC pauses, ECN marks)"]
  F --> P["Placement (which pods share which leaf / rail)"]
  G --> D["Data-loader wait (framework metrics)"]
  D --> S["Storage throughput (file-system metrics)"]
  • GPU idle and long NCCL time and rising PFC pauses on one leaf points to congestion, not GPUs.
  • GPU idle with short NCCL time but long data-loader wait points to storage.
  • One rank consistently late in every collective points to that rank's node: its clocks, its NIC, or its PCIe placement.

The observability module builds this correlation as dashboards.

Principles#

  1. Start from the workload. The model and its parallelism define the traffic; design the fabric and placement for that traffic.
  2. Design for the slowest rank. Remove variance (thermal, NUMA, congestion), not just averages.
  3. Validate at every layer, bottom up. GPU health, then NVLink, then RDMA point-to-point, then nccl-tests on the real placement, then a profiled training step.
  4. Make placement explicit. Topology labels, gang scheduling and rank mapping are part of the design, not defaults.
  5. Measure in production. Correlated telemetry across layers is how regressions are caught after day one.