Skip to content

L9 · Accelerator and Memory#

conceptstub

What it is#

The processors that do the math (NVIDIA and AMD GPUs, Google TPUs and other accelerators) with their compute units (SMs, tensor cores) and high-bandwidth memory (HBM).

Why it matters#

HBM capacity decides how much of a model, its optimizer state and its KV cache fit on each device, which drives the parallelism choice above. Memory bandwidth bounds inference token rates, and features such as MIG partitioning and FP8/FP4 precision shape how the hardware is shared and used.

In this module#

Where the computation actually happens.

Planned content

  • SMs, tensor cores, HBM, and memory bandwidth
  • Generations: H100, H200, B200, GB200
  • Hardware partitioning (MIG) and confidential computing mode