Online Inference

conceptstub

Serving models behind latency SLOs.

Planned content

• TP within a node for latency
• KV-cache memory and batching
• Autoscaling: where HPA/KEDA do fit