Skip to content

Sequence and Context Parallel#

concept

Long-context training breaks a different limit from model size: activation memory grows with sequence length, and attention compute grows with its square. Context parallelism (CP) splits the sequence itself across GPUs, for every layer.

Two things called "sequence parallel"#

Name What it splits Where Purpose
Sequence parallel (SP), Megatron Activations of layer norm and dropout Only the regions between TP blocks Companion to TP; saves activation memory
Context parallel (CP) The full sequence, in all layers including attention Whole model Long contexts (32K to 1M+ tokens)

SP is covered on the tensor parallel page. This page is about CP.

How context parallel works#

Most of a transformer layer (MLP, projections, norms) works on each token independently, so each GPU can process its own chunk of the sequence. Attention is the exception: every query needs the keys and values of other tokens.

Ring attention solves this. Each GPU keeps its chunk of queries and passes its key/value blocks around a ring:

step 0: GPU0 has K/V0   GPU1 has K/V1   GPU2 has K/V2   GPU3 has K/V3   (each computes attention on local block)
step 1: GPU0 gets K/V3  GPU1 gets K/V0  GPU2 gets K/V1  GPU3 gets K/V2  (send next while computing current)
step 2: …                                                                 (after N−1 steps every Q has seen every K/V)

The send of the next block overlaps with attention compute on the current block, so with a fast enough link the communication is hidden. With causal masking, the sequence is split into chunks assigned in a balanced pattern so every GPU gets a similar amount of work.

An alternative: all-to-all on heads#

DeepSpeed Ulysses uses all-to-all to switch from "split by sequence" to "split by attention heads" around the attention block, then switches back. Its degree is limited by the number of attention heads, and all-to-all stresses the network like expert parallelism does.

Communication profile#

  • Volume grows with sequence length and with the number of K/V heads.
  • Ring exchanges happen in every attention layer, forward and backward.
  • The degree of overlap depends on attention compute per block. Short chunks on fast GPUs leave less time to hide the transfer.

Placement#

CP sits just outside TP in the placement order: inside the NVLink domain when possible, or across a small number of nodes with a high-bandwidth fabric.

When to use#

  • Training or fine-tuning with contexts long enough that activations do not fit even with TP, SP and activation recomputation.
  • Typically combined: TP × CP inside a node or rack, then PP and DP outside.