Skip to content

L5 · Communication Libraries#

conceptstub

What it is#

Libraries that move data between GPUs for distributed work: NCCL (NVIDIA) or RCCL (AMD) for collectives such as all-reduce and all-gather, MPI for HPC, and NIXL for point-to-point transfers such as KV-cache movement in inference.

Why it matters#

Every distributed step synchronizes through these libraries. They pick algorithms (ring, tree) and transports (NVLink, RDMA, or a slow TCP fallback) automatically, so a misconfiguration silently costs performance rather than failing. They are the bridge between the parallelism strategy above and the hardware paths below.

In this module#

Collective and point-to-point communication for distributed GPU work.

Planned content

  • NCCL, MPI, and NIXL for KV-cache transfer
  • Algorithm and protocol selection
  • Container packaging and version alignment with the host