Glossary#
Parallelism and training#
- All-gather
- Collective where every rank ends up with the concatenation of every rank's data.
- All-reduce
- Collective where every rank ends up with the sum (or other reduction) of every rank's data. Equivalent to reduce-scatter followed by all-gather.
- All-to-all
- Collective where every rank sends a distinct piece of data to every other rank. Used by expert parallelism and Ulysses-style sequence parallelism.
- Bubble
- Idle time in pipeline parallelism while stages wait for micro-batches to arrive or drain.
- CP (context parallel)
- Splitting the sequence across GPUs in all layers, with ring exchange of keys and values in attention.
- DP (data parallel)
- Replicating the model and splitting the batch; gradients are all-reduced.
- EP (expert parallel)
- Placing Mixture-of-Experts experts on different GPUs; tokens are routed with all-to-all.
- FSDP
- Fully Sharded Data Parallel (PyTorch). Data parallelism that shards parameters, gradients and optimizer state across the DP group.
- HSDP
- Hybrid sharding: FSDP within a group, replication across groups.
- Micro-batch
- A slice of the global batch processed as one unit by a pipeline stage.
- MoE (Mixture of Experts)
- Model architecture where a router sends each token to a few of many expert sub-networks.
- PP (pipeline parallel)
- Splitting the model's layers into stages on different GPUs.
- Reduce-scatter
- Collective where the reduced result is split so each rank keeps one shard.
- SP (sequence parallel)
- In Megatron, splitting activations of non-tensor-parallel regions (norms, dropout) along the sequence dimension.
- TP (tensor parallel)
- Splitting each layer's weight matrices across GPUs.
- ZeRO
- DeepSpeed's family of optimizer-state (stage 1), gradient (stage 2) and parameter (stage 3) sharding.
Inference and agents#
- Agent
- An application that uses a model to plan and take steps, calling tools between model calls.
- MCP (Model Context Protocol)
- Open protocol that connects models and agents to tools and data through MCP servers.
- Decode
- Generating output tokens one at a time; usually bound by memory bandwidth.
- KV cache
- Stored attention keys and values for tokens already processed, reused for each new token.
- Prefill
- Processing the input prompt in one pass; usually compute-bound.
Communication and networking#
- DCQCN
- Data Center Quantized Congestion Notification. Congestion control for RoCE v2 driven by ECN marks.
- ECN
- Explicit Congestion Notification. Switches mark packets instead of dropping them when queues build.
- GPUDirect RDMA (GDR)
- NIC reads and writes GPU memory directly, without staging in host memory.
- InfiniBand (IB)
- Lossless, credit-based HPC fabric with native RDMA.
- libfabric (OFI)
- Open Fabrics Interfaces library that abstracts fabric providers (efa, verbs, cxi).
- NCCL
- NVIDIA Collective Communications Library; implements collectives over NVLink, PCIe, InfiniBand, RoCE and sockets.
- NIXL
- NVIDIA Inference Xfer Library; point-to-point data movement (e.g. KV cache) across memory and storage tiers.
- PFC
- Priority Flow Control. Per-priority pause frames that make Ethernet lossless for RoCE.
- Rail-optimized topology
- Fabric where GPU i of every node connects to the same leaf switch ("rail i").
- RDMA
- Remote Direct Memory Access. One host reads or writes another's memory without involving the remote CPU or the kernel data path.
- RoCE v2
- RDMA over Converged Ethernet, routable over UDP/IP.
- Verbs
- The low-level RDMA programming interface (
libibverbs).
GPU and node#
- HBM
- High Bandwidth Memory stacked on the GPU package.
- MIG
- Multi-Instance GPU. Hardware partitioning of a GPU into isolated instances.
- MPS
- Multi-Process Service. Lets kernels from several processes run concurrently on one GPU.
- NUMA
- Non-Uniform Memory Access. CPU sockets each have local memory and PCIe devices.
- NVLink / NVSwitch
- NVIDIA's GPU-to-GPU interconnect and the switch that connects all GPUs in a node (or rack) at full bandwidth.
- SM
- Streaming Multiprocessor, the GPU's basic compute unit.
Kubernetes#
- CDI
- Container Device Interface. A standard spec describing how to inject devices into containers.
- Device plugin
- Kubelet extension that advertises countable devices such as
nvidia.com/gpu. - DRA
- Dynamic Resource Allocation. Structured device claims (
ResourceClaim) with attributes and sharing, replacing simple device counts. - Gang scheduling
- Scheduling all pods of a job together, or none of them.
- Multus
- Meta-CNI that attaches multiple network interfaces to a pod.
- NetworkAttachmentDefinition (NAD)
- Multus custom resource describing a secondary network.
- Topology Manager
- Kubelet component that aligns CPU, memory and device allocations to NUMA nodes.