Disaggregated Inference#
conceptstub
Separate prefill and decode pools with KV-cache transfer.
Planned content
- Why split prefill and decode
- KV transfer over RDMA (NIXL)
- Routing and scheduling across pools
Separate prefill and decode pools with KV-cache transfer.
Planned content