Push-based KV transfer reduces TTFT and TPOT in vLLM disaggregated inference
· Engineer's Notes · Cem Koyluoglu
Push-mode NIXL connector cuts time‑to‑first‑token up to 3× and improves token latency by 30% for Qwen3‑32B on EC2 P5en with EFA, especially at high request rates.
What happened
vLLM separates LLM inference into a prefill phase that builds a key‑value (KV) cache and a decode phase that generates tokens from that cache. In a prefill‑decode (P‑D) disaggregated deployment, the KV cache must be moved from the prefill instance to the decode instance before token generation can start. The default NIXL connector uses a pull‑based approach: after prefill finishes, the router relays block IDs to the decode side, which then allocates destination blocks, prepares NIXL descriptors, and issues a READ to pull the KV blocks. This sequential coordination adds latency to the time‑to‑first‑token (TTFT) critical path.
The push‑based NIXL connector moves the coordination and setup to the prefill side. The decode instance registers its target blocks during allocation, allowing the prefill instance to start the transfer as soon as its KV blocks are ready. A dedicated background thread on the prefill worker prepares NIXL descriptors and issues a WRITE directly to the decode memory via RDMA (GPUDirect or Neuron Direct). Completion notifications inform the decode instance when all expected blocks have arrived, after which decoding begins.
Benchmarks on a 1P1D deployment using Amazon EC2 P5en instances with EFA show that, for the Qwen3‑32B model, push mode delivers up to 3× lower TTFT (≈540 ms vs. ≈1,744 ms) and up to 30 % lower time‑per‑output‑token (TPOT) (≈7 ms vs. ≈10.4 ms) at high request rates (4–8 QPS) and long prompts (up to 16 k tokens). At low load and short prompts, both modes perform similarly (TTFT around 540 ms, TPOT 5.6–5.8 ms). The benchmark disables prefix caching to force full KV transfer for each request.
Why it matters in production
The push‑based design directly addresses latency bottlenecks that appear when scaling inference workloads across separate prefill and decode nodes. By overlapping decode‑side block registration with prefill computation and offloading descriptor preparation to a background thread, the critical path for the first token is shortened. This reduction in TTFT improves perceived responsiveness for interactive applications such as chat or code completion, where users expect sub‑second replies.
Lower TPOT translates to higher token‑per‑second throughput on the decode side, enabling more concurrent requests without additional hardware. The approach also keeps the decode engine’s main loop free of blocking NIXL calls, which can improve overall system stability under bursty traffic. Because the transfer uses RDMA (GPUDirect or Neuron Direct) over AWS Elastic Fabric Adapter, data moves directly between device memories, minimizing host‑CPU involvement and reducing CPU load and memory bandwidth consumption.
From a cost perspective, the ability to achieve comparable or better throughput with the same instance types (P5en) can defer the need for larger clusters. The architecture remains accelerator‑agnostic, allowing deployment on both GPU and Trainium instances, which broadens hardware flexibility while preserving latency gains.
Engineering takeaways
- Register decode‑side KV blocks during allocation and initiate transfer from the prefill side to overlap work and eliminate post‑prefill round‑trips.
- Use a dedicated background thread on the prefill worker for NIXL descriptor preparation and WRITE submission, keeping the main inference loop free of blocking operations.
- Deploy on instances with RDMA‑capable interconnects (e.g., AWS EFA with GPUDirect or Neuron Direct) to enable direct device‑to‑device KV movement without host staging.
- Disable prefix caching in performance testing when the goal is to measure pure KV transfer latency, ensuring comparable workloads across pull and push modes.
- Configure both prefill and decode servers to use
NixlPushConnectorinstead of the defaultNixlConnectorto activate the push‑based flow.
Related work on this site
- RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
- Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
- Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.
This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.