Amazon SageMaker HyperPod Introduces Disaggregated Prefill and Decode for LLM Inference Optimization
Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD) to optimize large language model inference by splitting the prefill and decode phases onto dedicated GPU pools.
AWS has updated Amazon SageMaker HyperPod with support for Disaggregated Prefill and Decode (DPD). Large language model (LLM) inference consists of two distinct phases: the compute-heavy prefill phase and the memory-bandwidth-bound decode phase. By separating these phases onto dedicated GPU instances, DPD minimizes resource contention and improves overall latency and throughput.
Primary source
Open AWS What's New ↗This article summarizes the source. Check the original for full details and conditions.
Comparison
| Aspect | Before / Alternative | After / This |
|---|---|---|
| GPU Resource Allocation | A single GPU pool handles both compute-heavy prefill and memory-bound decode phases simultaneously. | Dedicated, isolated GPU pools are allocated independently for prefill and decode workloads. |
| Key-Value Cache Handling | KV caches are stored locally within the same GPU memory throughout the entire inference lifecycle. | KV caches are transferred across specialized node clusters to link prefill and decode stages. |
| Latency Predictability | Frequent latency spikes occur due to scheduling contention between prefill and decode operations. | Decoupled execution paths ensure predictable latency and highly optimized resource utilization. |