Workload Taxonomy#

concept

"AI workload" covers jobs with almost opposite infrastructure needs. A pre-training run wants hundreds of whole GPUs that start together and talk constantly. A RAG pipeline wants a slice of one GPU and a fast vector database. Designing one platform for both starts with naming the differences.

Four families#

$ tree ai-workloads/
ai-workloads/
├── training/
│   ├── pre-training
│   ├── full-fine-tuning
│   ├── peft-lora
│   └── rl-post-training
├── serving/
│   ├── online-inference
│   ├── disaggregated-inference
│   └── batch-inference
├── pipelines/
│   ├── rag
│   └── agentic-ai-mcp
└── interactive/
    └── notebooks-and-dev

Profiles#

Workload	Scale	Parallelism	Bound by	Lifecycle	Scheduling model
Pre-training	100s–10,000s GPUs	TP+PP+DP (+CP, EP)	Fabric, collectives, failures	Weeks–months	Gang-scheduled batch, checkpoint/restart
Full fine-tuning	8–128 GPUs	FSDP/ZeRO-3, sometimes TP	GPU memory, intra-node bandwidth	Hours–days	Gang-scheduled batch
PEFT / LoRA	1–8 GPUs	None or DP	GPU memory	Minutes–hours	Queued batch; fits shared clusters
RL post-training	8–1,000s GPUs	Training parallelism + inference engines	Phase switching, weight sync	Days	Gang-scheduled, mixed roles
Online inference	1–8 GPUs per replica, many replicas	TP within node, replicas for scale	HBM bandwidth, KV-cache capacity	Long-running service	Deployments with autoscaling
Disaggregated inference	Pools of prefill and decode GPUs	TP/EP per pool	KV transfer between pools	Long-running service	Separate pools, KV-aware routing
Batch inference	Any	Replicas (DP)	Throughput, cost	Minutes–hours	Queued, preemptible
RAG	Fractions of GPUs plus CPU/memory	Replicas	Retrieval latency, vector store	Long-running service	Services plus fractional GPUs
Agentic AI (MCP)	Model replicas plus many small tool services	Replicas; TP within a node for the model	Chained-call latency, KV-cache reuse, tool waits	Long-lived sessions	Autoscaled services, prefix-aware routing
Notebooks / dev	Fractions of a GPU	None	Idle time, contention	Interactive	Fractional GPUs, idle reclamation

Training family#

Pre-training stresses every layer at once. It is the reason RDMA fabrics, NVLink domains, gang scheduling and parallel file systems exist. The defining operational problem is failure: at thousands of GPUs, a node fails every few hours, so checkpoint frequency and restart speed decide effective throughput.

Full fine-tuning updates all weights of an existing model on a smaller dataset. The memory footprint per parameter is the same as pre-training, so FSDP/ZeRO-3 is the default, but the scale is one to a few dozen nodes.

PEFT (LoRA, QLoRA) trains small adapter matrices on a frozen base model. Optimizer state is tiny, the base can be quantized, and many jobs fit on one GPU. These jobs are the best fit for shared, multi-tenant clusters.

RL post-training (RLHF, GRPO and similar) alternates between generating responses with an inference engine and training on the scored results. One job contains two very different workloads, and the trained weights must be synced to the inference engines every iteration. Frameworks either colocate both on the same GPUs and switch phases, or run them on separate GPU pools.

Serving family#

Online inference serves requests under latency targets. Two phases dominate: prefill (process the prompt; compute-bound) and decode (generate tokens one at a time; memory-bandwidth-bound). The KV cache grows with context length and concurrent requests and is usually what limits batch size. Unlike training, inference autoscales: replicas come and go with load, so HPA/KEDA-style scaling is appropriate.

Disaggregated inference runs prefill and decode on separate GPU pools so each can be sized and batched for its own bottleneck. The KV cache produced by prefill is transferred to a decode GPU, often over RDMA (for example with NIXL). This puts the fabric back on the critical path for serving.

Batch inference (offline scoring, evaluation, synthetic data) is throughput-oriented and tolerant of delay, which makes it a good consumer of preemptible and idle capacity.

Pipelines and interactive work#

RAG combines an embedding model, a vector store, optionally a reranker, and a generator LLM. Only the generator is large; the other models are small enough for fractional GPUs (MIG, HAMi), and the vector store is a CPU, memory and storage problem.

Agentic AI chains many model calls per task, with tools reached through MCP servers in between. The tool servers are ordinary web-style services; the pressure lands on online inference: per-call latency, KV-cache reuse across shared prompt prefixes, and autoscaling for long-lived sessions.

Notebooks and development hold GPUs for long periods while using them briefly. Fractional GPUs, quotas and idle reclamation matter more than raw performance.

Which layers each workload stresses#

workload \ layer	L1	L2	L3	L4	L5	L6	L7	L8	L9	S	CP
Pre-training	■■	■■■	■■■	■■■	■■■	■■■	■■■	■■■	■■■	■■■	■■■
Full fine-tuning	■■	■■■	■■	■■■	■■	■■	■■	■■■	■■■	■■	■■
PEFT / LoRA	■■	■■■	■	■■	■	·	·	■	■■	■	■■
RL post-training	■■■	■■■	■■■	■■■	■■■	■■	■■	■■■	■■■	■■	■■■
Online inference	■■■	■■■	■■	■■■	■	·	■	■■■	■■■	■	■■■
Disaggregated inference	■■■	■■■	■■	■■■	■■	■■■	■■■	■■■	■■■	■	■■■
RAG	■■■	■■	·	■	·	·	·	·	■	■■■	■■
Agentic AI (MCP)	■■■	■■■	■	■■	·	·	·	■■	■■	■	■■■
Batch inference	■	■■	■	■■	·	·	·	■	■■	■■	■■
Notebooks / dev	■	■	·	■	·	·	·	·	■	■	■■■

·low■moderate■■high■■■critical

Each row links to the workload page, which lists the parallelism, scheduling, networking, storage and GPU-sharing choices for that workload.