Multi-node LLM inference on Amazon EKS using Ray LLM Deep Learning Containers
· Engineer's Notes · Cem Koyluoglu
Deploy a large language model across two GPU nodes on Amazon EKS with Ray LLM DLC, separating prefill and decode phases for improved latency and throughput.
What happened
The AWS Machine Learning Blog post describes a deployment that runs a single large language model (LLM) across two GPU‑powered nodes in an Amazon Elastic Kubernetes Service (EKS) cluster. The architecture uses Ray Serve inside the Ray LLM Deep Learning Container (DLC) to host a prefill worker on one g6.8xlarge instance and a decode worker on a second g6.8xlarge instance. A CPU‑only m7i.xlarge node runs the Ray head and the KubeRay operator. Communication between the workers relies on the NVIDIA Inference Transfer Library (NIXL) and the Elastic Fabric Adapter (EFA) stack, which provides high‑bandwidth, low‑latency transfers of the key‑value (KV) cache built during the prefill phase. The deployment is defined in a YAML manifest that leverages the build_pd_openai_app helper to wire the prefill and decode configurations behind a single OpenAI‑compatible endpoint. The post includes step‑by‑step scripts for creating the EKS cluster, adding the GPU node group, installing KubeRay, and deploying the RayService. Model loading time for the Qwen3.5‑9B model drops from roughly 145 seconds to about 2 seconds on restart when the described cache configuration is used.
Why it matters in production
Separating the compute‑heavy prefill phase from the latency‑sensitive decode phase eliminates resource contention on a single GPU, allowing each phase to be tuned independently. This disaggregation improves overall throughput for workloads such as Retrieval Augmented Generation, document summarization, and agentic applications. The use of pre‑built Ray LLM DLCs removes the need for teams to manually align CUDA runtimes, framework versions, and serving layers, reducing undifferentiated engineering effort and the risk of version drift. Leveraging EFA’s libfabric backend ensures KV‑cache transfers occur at near‑network speeds, preserving the performance gains of disaggregation; without the kv_connector_extra_config: { backends: ["LIBFABRIC"] } setting, NIXL would fall back to TCP‑speed UCX, eroding latency benefits. Deploying both GPU workers in a single Availability Zone is required because EFA cannot cross AZs, a constraint that influences cluster topology and cost planning. The described approach also benefits from the DLC’s built‑in security patches applied at build time, simplifying compliance in regulated environments.
Engineering takeaways
- Use Ray LLM DLCs to obtain a validated stack of CUDA, framework, and serving components, eliminating manual dependency management.
- Deploy prefill and decode phases on separate GPU nodes (e.g., two g6.8xlarge instances) and configure NIXL with the LIBFABRIC backend to achieve high‑speed KV‑cache transfer.
- Pin the GPU node group to a single Availability Zone to satisfy EFA’s AZ limitation and ensure consistent low‑latency interconnect.
- Set
max_model_len: 4096,dtype: bfloat16, andgpu_memory_utilization: 0.90to fit the KV cache within GPU memory while maintaining precision. - Allocate sufficient vCPU quota (at least 64 vCPU) and enable EFA on the GPU instances to meet the network requirements of multi‑node inference.
Related work on this site
- RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
- Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
- Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.
This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.