L2 · Frameworks and Serving Engines

conceptstub

What it is

Software that turns a model definition into GPU work. Training frameworks (PyTorch, JAX, Megatron-LM, DeepSpeed) build and run the computation, and manage gradients and optimizer state. Serving engines (vLLM, SGLang, TensorRT-LLM) run trained models for inference, managing request batching and the KV cache.

Why it matters

The framework is where parallelism is configured and where most performance features live: mixed precision, activation recomputation, overlap of communication with compute, continuous batching, paged attention. Its version also pins the CUDA, NCCL and driver versions that every layer below must match.

In this module

Frameworks that turn models into GPU work.

Planned content

• Training: PyTorch, JAX, Megatron-LM, DeepSpeed, NeMo
• Serving: vLLM, SGLang, TensorRT-LLM, Triton, Dynamo
• Container images, CUDA compatibility, and pinning