Run multi-node inference workloads using Ray LLM Deep Learning Containers

This post shows how to deploy one large language model (LLM) across two GPU nodes on Amazon Elastic Kubernetes Service (Amazon EKS) with the multi-node Ray LLM Deep Learning Containers. It uses separate prefill and decode nodes behind one OpenAI-compatible endpoint. This kind of multi-node setup is common for tensor, pipeline, and data parallelism, as well as prefill/decode disaggregation inference workloads. In a previous post, we introduced the Ray Serve DLC and used it to stand up a small model behind an HTTP endpoint on a single GPU node. You had a working inference endpoint, and the container keeps the GPU stack compatible, including CUDA libraries, so you did not need to write boilerplate code.

Ray is a distributed compute framework, so Ray Serve deployments can span multiple nodes. Customers use multi-node Ray Serve for several reasons. Some want to run foundation or open-weight models that are too large for a single GPU. Others need more throughput than one replica can deliver. Still others want to separate the two phases of inference to improve performance. In each case, customers must build the dependency chain themselves: choosing compatible libraries for the GPU stack and debugging issues when a component drifts out of alignment. This is undifferentiated work that takes time and effort away from delivering models.

AWS Deep Learning Containers (DLCs) have long solved this kind of problem for training and inferencing workloads. DLCs are pre-built, performance-optimized Docker images that bundle a framework, its dependencies, and the GPU stack into a tested, patched combination you can pull and use immediately. With the launch of the Ray LLM DLC for multi-node deployments, that same approach now extends to multi-node inference. You get a container purpose-built for serving models behind an HTTP endpoint, maintained and tested by AWS, with the full inference stack already assembled.

This post walks through implementing multi-node disaggregated inference using multi-node Ray LLM DLC: deploying a single large language model across two GPU-powered nodes on Amazon EKS cooperating behind a single endpoint. While one of the GPU nodes handles the prefill phase, the other GPU node handles the decode phase. Separating the phases prevents compute-heavy long prompts from stalling token generation for requests already in progress. This approach is well suited for workloads such as Retrieval Augmented Generation (RAG), document summarization, and agentic applications. We use the multi-node Ray LLM DLC, which ships Ray Serve LLM, NVIDIA Inference Transfer Library (NIXL), and the Elastic Fabric Adapter (EFA) stack, and exposes an OpenAI-compatible API with no application code. The complete code is available in the accompanying repository.

Why prefill/decode (PD) disaggregation

LLM inference has two distinct phases. Prefill processes the entire input prompt in parallel and builds the key-value (KV) cache. Decode then generates the response one token at a time, reading and extending that cache. Prefill is compute bound, and Decode is bound by memory bandwidth. When both phases share one GPU, they contend for the same resources, and the bursty, compute-heavy prefill work interferes with the steady, latency-sensitive decode work. Separating the prefill phase from the decode phase means you can select the hardware and serving properties for each phase, improving overall performance at scale.

PD disaggregation places the two phases on separate nodes, so each phase runs without interfering with the other and can be scaled independently. The trade-off is that the KV cache built during prefill must be handed to the decode node, and that transfer should be fast enough not to erase the benefit. That is exactly why the interconnect between the nodes matters.

What the Ray LLM DLC provides

AWS DLCs are pre-built, performance-optimized Docker images that bundle a framework, its dependencies, and the GPU stack into a tested, patched combination you can pull and use immediately. All components are validated together before each release, so there is no version drift between the CUDA runtime, the framework, and the serving layer. Security patches are applied at build time.

The Ray LLM DLC extends this to distributed LLM serving. It ships vLLM as the inference engine, Ray Serve’s HTTP as the serving layer, NIXL for KV-cache transfer between nodes, and the EFA stack (libfabric and aws-ofi-nccl) for high-bandwidth node-to-node communication. Because the Ray Serve LLM library includes a built-in application builder, the entire serving application is a block of YAML. There’s no handler class, no model archiver, and no custom image to build. A single helper, build_pd_openai_app, takes a prefill configuration and a decode configuration and wires the KV-cache transfer between them behind one OpenAI-compatible endpoint.

Architecture

The following diagram shows how the deployment is laid out on Amazon EKS. A user request reaches the Ray head on the CPU system node, and the two GPU worker nodes run the prefill and decode phases of the model.

Figure 1: Multi-node inference architecture on Amazon EKS, with the Ray head on a CPU system node and the prefill and decode phases on two GPU worker nodes

The deployment is an EKS cluster inside a virtual private cloud (VPC). Nodes run in private subnets with outbound access through a NAT Gateway. A CPU system node (m7i.xlarge) hosts system workloads, the KubeRay operator, and the Ray head pod, which is CPU-only. Two GPU worker nodes (g6.8xlarge) run the Ray workers: one serving the prefill phase, and the other serving the decode phase. Both nodes pull the Ray LLM DLC from Amazon Elastic Container Registry (Amazon ECR).

KubeRay watches the RayService you apply and wires the head-to-worker join automatically. Ray Serve places the prefill and decode deployments on the two GPU workers. The KV cache flows from the prefill worker to the decode worker over NIXL, using EFA through NIXL’s libfabric backend. The GPU node group is pinned to a single Availability Zone (AZ) because EFA cannot cross AZs.

Prerequisites

To follow along with this post, you need:

• An AWS account with billing enabled.
• Sufficient service quota for running two g6.8xlarge instances (at least 64 vCPU) in your target Region.
• AWS Command Line Interface (AWS CLI) with credentials configured.
• kubectl CLI for communicating with a Kubernetes cluster’s control plane.
• eksctl CLI for creating and managing clusters on EKS.
• helm (package manager for Kubernetes) to install KubeRay operator.
• envsubst utility (part of the GNU gettext) to render the manifest.

Verify that your AWS credentials are active with the following command:

aws sts get-caller-identity

Use the pre-built application to serve inference

The application is the serveConfigV2 block in manifest/rayservice.yaml. It uses build_pd_openai_app, which takes a prefill_config and a decode_config and connects the KV-cache transfer between them:

applications:
- name: qwen
  import_path: ray.serve.llm:build_pd_openai_app # OpenAI-compatible PD app, built in
  route_prefix: /
  args:
    prefill_config:
      model_loading_config: { model_id: qwen3.5-9b, model_source: Qwen/Qwen3.5-9B }
      engine_kwargs:
        max_model_len: 4096
        dtype: bfloat16
        gpu_memory_utilization: 0.90
        enforce_eager: true # Skip CUDA graph capture to reduce peak GPU memory
      kv_transfer_config:
        kv_connector: NixlConnector
        kv_role: kv_both
        engine_id: prefill
        kv_connector_extra_config: { backends: ["LIBFABRIC"] }
      deployment_config: { num_replicas: 1 }
    decode_config:
      model_loading_config: { model_id: qwen3.5-9b, model_source: Qwen/Qwen3.5-9B }
      engine_kwargs:
        max_model_len: 4096
        dtype: bfloat16
        gpu_memory_utilization: 0.90
        enforce_eager: true
      kv_transfer_config:
        kv_connector: NixlConnector
        kv_role: kv_both
        engine_id: decode
        kv_connector_extra_config: { backends: ["LIBFABRIC"] }
      deployment_config: { num_replicas: 1 }

A few settings carry most of the weight. max_model_len: 4096 sizes the context to fit the measured KV cache on the GPU. dtype: bfloat16 and gpu_memory_utilization: 0.90 set precision and how much VRAM vLLM can claim. enforce_eager: true skips CUDA graph capture to reduce peak GPU memory (model- and input-dependent). The kv_transfer_config setting is what makes disaggregation work: both phases use NixlConnector with kv_role: kv_both and a distinct engine_id (prefill on one, decode on the other). The critical setting is kv_connector_extra_config: { backends: ["LIBFABRIC"] }. Without that line, NIXL defaults to UCX, which runs at TCP speed even on EFA.

The rest of the manifest, in rayClusterConfig, describes the pods. The head pod is CPU-only (num-gpus: "0") and pinned to the system node group. Each worker pod requests one GPU and one EFA interface (nvidia.com/gpu: "1", vpc.amazonaws.com/efa: "1") and targets the GPU nodes through nodeSelector: { role: gpu-worker }. It runs privileged: true because NIXL registers CUDA VRAM with the EFA NIC. The IPC_LOCK capability alone isn’t enough. The workers set FI_PROVIDER=efa, read VLLM_HOST_IP from the pod IP, and mount a shared-memory volume. Memory limits sit well above the roughly 18 GiB checkpoint so the OS page cache can hold it. For Qwen3.5-9B model loading on g6.8xlarge with the preceding cache configuration, the model weights loading drops from around 145 seconds to about 2 seconds on restart.

Deploy on Amazon EKS

All scripts share a single configuration file, scripts/env.sh. The script contains defaults for CLUSTER_NAME, REGION, K8S_VERSION, and NAMESPACE. Other defaults cover the system and GPU node groups, the KubeRay version, and the model. Override any variable by exporting it before running a script.

From the project root, set a base directory, then load the variables:

CURRENT_DIR=$(pwd)
cd $CURRENT_DIR/ray-serve-multi-node/scripts
source ./env.sh

Step 1: Create the EKS cluster

./deploy_cluster.sh

This provisions the EKS cluster (VPC, OIDC provider, core add-ons) and a CPU system node group that runs system workloads, the KubeRay operator, and the Ray head. Nodes run in private subnets with egress through a NAT Gateway, and cluster subnets are placed in the AZs that offer the GPU instance type. The script is idempotent and takes 15–20 minutes on a fresh run.

Step 2: Add GPU worker nodes

./deploy_node_group.sh

This creates the GPU node group: two g6.8xlarge instances with EFA enabled, pinned to a single AZ and labeled role=gpu-worker so the Ray workers target them. Enabling EFA also installs the EFA device plugin, which advertises vpc.amazonaws.com/efa on each node. This step takes about 5-20 minutes, usually longer on a first run.

Step 3: Install the KubeRay operator

./install_kuberay.sh

This installs the KubeRay operator using Helm, pinned to the system node group to keep the GPU nodes available for inference. KubeRay watches the RayService you apply next and wires the head-to-worker join automatically. This step takes about 1–2 minutes.

Step 4: Deploy the RayService

./deploy_ray_service.sh

This renders manifest/rayservice.yaml with your image, model, and naming variables using envsubst and applies it. KubeRay creates the head and two worker pods, and Ray Serve LLM brings up the prefill and decode deployments. The script waits for the head to be Ready, both workers to be Running, and finally for the Serve app to report RUNNING. Because each node loads the model, expect this step to take several minutes.

Check status at any time, including which node each pod landed on and the GPU capacity available:

./deploy_ray_service.sh status

Invoke the model

After the Serve app is healthy, KubeRay creates a stable <rayservice-name>-serve-svc Service. Port-forward to it:

kubectl port-forward -n inference svc/ray-llm-serve-svc 8000:8000

Because the DLC exposes an OpenAI-compatible API, you can list models and send a standard chat-completion request with no custom schema:

curl --silent http://127.0.0.1:8000/v1/models
curl --fail --silent --show-error 
  --request POST "http://127.0.0.1:8000/v1/chat/completions" 
  --header "Content-Type: application/json" 
  --data '{
    "model": "qwen3.5-9b",
    "messages": [
      {"role": "user", "content": "Explain the benefits of prefill/decode disaggregation."}
    ],
    "max_tokens": 200,
    "temperature": 0.7
  }'

Verify EFA is being used

NIXL falls back to a TCP-speed transport if it cannot use libfabric, and it does so without raising an error, so it’s worth confirming the backend it picked. Grep the worker logs:

for POD in $(kubectl get pods -n inference -l ray.io/node-type=worker 
  -o jsonpath='{.items[*].metadata.name}'); do
  kubectl exec -n inference "$POD" -c ray-worker -- 
    grep -rhoE "Backend (LIBFABRIC|UCX) was instantiated" /tmp/ray/session_latest/logs/ 2>/dev/null
done | sort | uniq -c

deploy_node_group.sh already prints the vpc.amazonaws.com/efa count per node when it creates the group. For a cross-node bandwidth number, the DLC ships a prebuilt NVIDIA Collective Communications Library (NCCL) benchmark at /usr/local/bin/all_reduce_perf.

Clean up

To avoid ongoing charges, tear down the resources in reverse order:

./delete_ray_service.sh # Remove the RayService
./install_kuberay.sh cleanup # Uninstall the KubeRay operator
./delete_node_group.sh # Remove the GPU node group
./delete_cluster.sh # Delete the EKS cluster

Conclusion and next steps

You served a single model across two GPU nodes with prefill/decode disaggregation, and the entire application was a block of YAML, with no handler code, no model archiver, and no hand-assembled GPU and EFA stack. The multi-node Ray LLM DLC supplied the tested inference stack, KubeRay orchestrated the Ray cluster, and NIXL over EFA carried the KV cache between nodes. Your job was to configure it, not to build it.

Common patterns for running multi-node inference include tensor and pipeline parallelism (sharding a model too big for one instance), data parallelism (replicating a model for throughput), and prefill/decode disaggregation (separating the two phases of inference onto different nodes).

From here you can scale the decode replicas for more throughput, move to larger models with tensor or pipeline parallelism on larger GPU instances, or layer extra libraries onto the same tested base for models that need them. For the full code, the current list of image tags, and the underlying mechanics, see the accompanying repository, the Ray LLM DLC available images page, the Ray Serve LLM prefill/decode documentation, and the EFA documentation.