Skip to content

From Pod Spec to Data Path#

concept

How a training pod gets its GPUs and its RDMA NIC, step by step. This is the control plane at work: it decides where the pod runs, attaches the right devices and networks, and then steps out of the way. Once the pod is running, GPU traffic never passes through Kubernetes again.

The sequence#

sequenceDiagram
    autonumber
    actor U as User
    participant API as API server
    participant Q as Kueue
    participant SCH as Scheduler
    participant KL as Kubelet
    participant DP as Device plugins (GPU, RDMA)
    participant CRI as containerd + NVIDIA toolkit
    participant CNI as Multus (Calico, SR-IOV CNI)
    participant POD as Pod (NCCL)

    U->>API: Submit job (8 GPUs + 1 RDMA NIC per pod)
    API->>Q: Workload created, held until admitted
    Q->>Q: Check quota, admit all pods together (gang)
    Q->>SCH: Release pods for scheduling
    SCH->>SCH: Filter nodes with free GPUs and NICs, score by topology
    SCH->>API: Bind each pod to a node
    API->>KL: Pod assigned to this node
    KL->>KL: Topology Manager: pick GPUs and NIC on the same NUMA node
    KL->>DP: Allocate GPU IDs and an RDMA device (SR-IOV VF)
    DP-->>KL: Device nodes, environment, CDI specs
    KL->>CRI: Create pod sandbox
    CRI->>CNI: CNI ADD for the pod network namespace
    CNI->>CNI: eth0 from Calico (pod network, services, DNS)
    CNI->>CNI: net1 from SR-IOV CNI (the allocated VF, RDMA-capable)
    KL->>CRI: Create containers with /dev/nvidia* and /dev/infiniband
    CRI->>POD: Start containers
    POD->>POD: NCCL finds GPUs, uses net1 for RDMA, eth0 for bootstrap
    Note over KL,POD: From here on, data flows GPU, NVLink, PCIe, NIC, fabric. The control plane is out of the path.

The pod spec that drives it#

apiVersion: v1
kind: Pod
metadata:
  name: trainer-0
  labels:
    kueue.x-k8s.io/queue-name: team-a # (1)!
  annotations:
    k8s.v1.cni.cncf.io/networks: rdma-net-rail0 # (2)!
spec:
  containers:
    - name: trainer
      image: nvcr.io/nvidia/pytorch:<tag>
      securityContext:
        capabilities:
          add: ["IPC_LOCK"] # (3)!
      resources:
        limits:
          nvidia.com/gpu: 8 # (4)!
          nvidia.com/sriov_rdma_rail0: 1 # (5)!
      env:
        - name: NCCL_IB_HCA
          value: "mlx5" # (6)!
        - name: NCCL_SOCKET_IFNAME
          value: "eth0" # (7)!
  1. Control plane: Kueue holds the pod until the team's queue has quota, and admits all pods of the job together so none sit idle waiting for the others.
  2. Control plane → L7: Multus reads this annotation and attaches a second interface, net1, from the NetworkAttachmentDefinition named rdma-net-rail0. Calico still provides eth0.
  3. L6: RDMA pins (registers) memory so the NIC can read it directly. Without IPC_LOCK, memory registration fails and NCCL falls back to TCP.
  4. Control plane → L9: The GPU device plugin (deployed by the GPU Operator) allocates 8 GPUs and exposes them to the container.
  5. Control plane → L6/L7: The SR-IOV device plugin allocates one RDMA-capable virtual function. The resource name is whatever your SR-IOV configuration defines.
  6. L5: Tells NCCL which RDMA devices to use for GPU traffic.
  7. L5: NCCL's bootstrap (rank discovery) uses plain TCP over the pod network, not the RDMA interface.

What can go wrong at each step#

Step Failure What you see
Admission No quota, or the gang cannot fit Pods stay Pending; Kueue shows the workload as not admitted
Scheduling No node has 8 free GPUs and a free VF 0/N nodes are available: Insufficient nvidia.com/sriov_...
Topology Manager GPUs and NIC on different NUMA nodes With policy single-numa-node: TopologyAffinityError. With best-effort: the pod runs, but slowly
Device plugin Driver or plugin not ready Insufficient nvidia.com/gpu although the GPUs are physically present
CNI Bad NetworkAttachmentDefinition or IPAM exhausted Pod stuck in ContainerCreating, CNI error in the pod's events
Runtime Pod is running, but NCCL logs NET/Socket RDMA device or IPC_LOCK missing: training runs over TCP, many times slower

The last row is the dangerous one: nothing fails, the job just runs slowly. The slow collectives decision tree covers how to catch it.

With DRA instead of device plugins#

With Dynamic Resource Allocation (DRA), the pod references ResourceClaims instead of counting resources in limits. The scheduler allocates specific devices while it picks the node, so topology rules (same PCIe switch, same NVLink domain) are decided during scheduling rather than after it. The rest of the sequence is the same.