Close Menu
MyAppsPlus

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    Whisker offers up to $200 Litter-Robot savings during Black Friday Sale starting from $564

    November 18, 2026

    Motorola Razr Fold finally gets its Android 17 update

    November 18, 2026

    Google Store’s Black Friday coupons are in the mail

    November 18, 2026
    Facebook X (Twitter) Instagram
    Facebook X (Twitter) Instagram
    MyAppsPlusMyAppsPlus
    Wednesday, November 18
    • Home
    • Breaking Tech
    • Apps & Software
    • AI & Automation
    • Android
    • iPhone & iOS
    • More
      • Reviews
      • How-To Guides
      • Deals & Discounts
      • Shop
    MyAppsPlus
    Home»AI & Automation»Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
    AI & Automation

    Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

    myappsplusBy myappsplusNovember 18, 2026008 Mins Read
    Share Facebook Twitter Pinterest Copy Link LinkedIn Tumblr Email Telegram WhatsApp
    Follow Us
    Google News Flipboard
    Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
    Share
    Facebook Twitter LinkedIn Pinterest Email Copy Link

    Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.

    Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.

    How the integration works

    The HyperPod Inference add-on installs the Dynamo custom resource definitions and these serving and scheduling components:

    • HyperPod Inference Operatormanages endpoint deployment through an InferenceEndpointConfig (IEC) custom resource. Its dynamoSpec section supplies the Dynamo configuration.
    • Dynamo operatormanages resources defined in a DynamoGraphDeployment (DGD), including the frontend, model workers, replica counts, and resource requirements.
    • Grovecoordinates related groups of pods, including multi-node workers, and their scaling relationships.
    • Kubernetes AI (KAI) Schedulerplaces pods on GPU resources, using gang scheduling to schedule the pods required for a multi-node worker together.

    AWS Deep Learning Containers (DLCs) provide the Dynamo runtime, while the add-on manages integrated component versions. To adopt a supported Dynamo release, update the add-on.

    Deployment architecture

    The following diagram shows the deployment paths, serving components, and infrastructure management for disaggregated inference on SageMaker HyperPod. The Dynamo frontend routes requests to prefill workers, which process prompts, and decode workers, which generate tokens. These separate pools allow independent scaling and tuning. Both pools load the full model weights, and KV cache transfers between them over EFA.

    Dynamo also supports aggregated serving, where prefill and decode run in the same worker. In either topology, workers can run on one instance or span multiple instances using tensor and pipeline parallelism. Dynamo’s KV-aware router considers reusable prefix KV cache and active worker load in both topologies. Prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness helps avoid overloading workers. For more details on the Dynamo architecture, see the Dynamo Knowledge Base overview.

    HyperPod provides node, GPU, and EFA health monitoring and manages node replacement, while the orchestration components recreate affected workers. Deployments configured through InferenceEndpointConfig with dynamoSpec support invocation through SageMaker endpoints.

    Figure 1: Deployment paths, serving components, and infrastructure management for NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

    Two ways to configure your deployment

    Whether you are new to HyperPod, new to Dynamo, or both, the add-on offers two deployment paths to fit your workflow:

    Bring-your-own DynamoGraphDeployment (DGD) lets existing Dynamo users continue using upstream documentation, recipes, and manifests, with the Dynamo operator managing the deployment. To bring an existing DGD to HyperPod, use a supported container image and configure storage, networking, and pod placement for your cluster.

    InferenceEndpointConfig with dynamoSpec lets existing HyperPod Inference users deploy a new Dynamo endpoint using the same configuration format and OpenAI-compatible interface they already know. For the inference engine, you can choose vLLM, SGLang, or TensorRT-LLM, with vLLM used by default. The operator also supplies defaults for KV-aware routing and replica counts.

    Both paths support single-instance and multi-node deployments. In either case, HyperPod manages node health and replacement. The table summarizes the responsibilities for each path.

    Bring-your-own DGD InferenceEndpointConfig with dynamoSpec
    You write Complete DGD manifest IEC containing dynamoSpec
    Best for Teams wanting direct control of the serving graph or reuse of existing DGDs Teams wanting managed endpoint setup through IEC
    Model weights You configure downloading or mounting the weights The operator stages weights from Amazon Simple Storage Service (Amazon S3) or Amazon FSx
    Endpoint, TLS, registration You configure and manage these resources Managed through the HyperPod Inference Operator
    Autoscaling You configure and manage scaling Kubernetes Event-driven Autoscaling (KEDA) through autoScalingSpec for aggregated and disaggregated deployments

    Deployment example: GLM-5.3

    The following excerpts use the InferenceEndpointConfig (IEC) deployment path with vLLM on ml.p5.48xlarge instances. Dynamo-specific settings belong under spec.dynamoSpec. Model

    Multi-node aggregated serving

    This configuration runs one model-worker replica across two instances (16 GPUs), with two frontend replicas for request handling. Setting dynamoSpec.frontend.routingMode to kv-aware turns on KV-aware routing.

    spec:
      replicas: 1
      dynamoSpec:
        modelServer: vllm
        frontend:
          replicas: 2
          routingMode: kv-aware
        aggregation:
          nodesPerReplica: 2
          tensorParallelism: 16

    Here, spec.replicas controls the number of model-worker replicas, while frontend.replicas controls the number of frontend replicas.

    Multi-node disaggregated serving

    Replace aggregation under spec.dynamoSpec with the following disaggregation block, keeping the preceding frontend configuration. Replica counts are set separately for prefill and decode. This example runs one prefill replica and two decode replicas, each spanning two instances, for a total of six instances.

    disaggregation:
      prefill:
        nodesPerReplica: 2
        tensorParallelism: 16
        replicas: 1
      decode:
        nodesPerReplica: 2
        tensorParallelism: 16
        replicas: 2

    Apply the deployment

    Apply the complete manifest with kubectl apply. The HyperPod Inference Operator makes the configured model weights available to the workers and generates the DGD. Grove and KAI Scheduler coordinate placement so that the pods required for each multi-node worker are scheduled together.

    After the SageMaker endpoint is ready and configured for chat completions, clients can send an OpenAI-compatible request payload through the SageMaker Runtime API:

    aws sagemaker-runtime invoke-endpoint 
    --endpoint-name glm53 
    --content-type application/json 
    --body '{"model":"zai-org/GLM-5.3","messages":[{"role":"user","content":"Hello"}],"max_tokens":128}' 
    --cli-binary-format raw-in-base64-out 
    out.json

    Benchmarks

    We benchmarked Dynamo on SageMaker HyperPod across two models to compare serving configurations.

    Disaggregated prefill and decode for Qwen3-235B

    We used AIPerf to compare aggregated and disaggregated serving for Qwen3-235B-A22B-FP8 using Dynamo with TensorRT-LLM on two p5en.48xlarge instances (16 H200 GPUs). Requests contained 4,000 input tokens and 200 output tokens. We retained measured points that met two targets: P90 time to first token below one second, and median generation speed of at least 50 tokens per second per user. Among these points, disaggregated serving achieved 2.05× the highest aggregated throughput: 207 versus 101 output tokens per second per GPU, at concurrency 80 and 32, respectively.

    Figure 2: Aggregated compared to disaggregated serving throughput for Qwen3-235B-A22B-FP8

    Multi-node aggregated serving for Qwen3.5-397B

    Dynamo supports multi-node serving in both aggregated and disaggregated modes for models that exceed a single instance’s GPU memory. To evaluate a worker spanning nodes, we used AIPerf to benchmark multi-node aggregated serving for Qwen3.5-397B-A17B-FP8 with Dynamo and vLLM, using TP4 × PP2 and eight active H200 GPUs. This model fits on one instance, so in practice you would not split it. It serves here as a controlled baseline. We compared eight GPUs on one p5en.48xlarge against four GPUs on each of two instances, using the same model, parallelism, and workload configuration. With an agentic workload featuring high prefix reuse at concurrency 32, the two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 percent of single-node throughput. This shows that Dynamo on SageMaker HyperPod can serve a model that needs more than one instance across nodes. The node boundary itself does not become a significant throughput penalty.

    Figure 3: Single-node compared to two-node throughput for multi-node aggregated serving of Qwen3.5-397B-A17B-FP8

    Best practices for optimized inference

    NVIDIA Dynamo provides recipes and tools to help you select a serving configuration, benchmark performance, and tune your deployment on SageMaker HyperPod.

    • Start with a matching recipe – Choose a Dynamo recipe for your model, inference engine, and GPU type. Account for model weights, KV cache, and runtime memory when sizing workers. For supported configurations, AISimulate can help compare aggregated and disaggregated serving, choose parallelism, and size prefill and decode worker pools before testing on GPUs.
    • Verify communication before benchmarking – For deployments spanning instances, confirm that communication uses the intended EFA transport, including KV-cache transfers between prefill and decode workers. See the Dynamo EFA guide.
    • Benchmark representative traffic – Replay your traffic with AIPerf, or use a generated agentic trace with representative prompt lengths, output lengths, and prefix reuse. Sweep concurrency for each configuration, retain points meeting your SLAs, and choose based on cost and latency priorities.
    • Tune from measured bottlenecks – Adjust topology and worker allocation before engine settings, changing one variable at a time. Follow the performance tuning guide, or use Dynamo optimization agent skills to assist with benchmarking and tuning. Validate each candidate on your target HyperPod deployment.

    Conclusion

    NVIDIA Dynamo on SageMaker HyperPod Inference simplifies deploying and operating large models with multi-node serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs the serving components, and configuration defaults simplify endpoint setup. HyperPod manages model weights, auto-scales clusters based on GPU and inference metrics, monitors node health, and replaces failed nodes. Together, these capabilities reduce undifferentiated infrastructure work so teams can focus on their models and applications. Get started with the setup instructions and complete deployment examples in the linked repository.

    Acknowledgements

    The authors would like to thank Eliuth Triana, Felipe Garcia, Sanjana Checker, Junyuan Yang, Jiahong Liu, Anish Maddipoti, Jacob Kern and Abdullahi Olaoye from NVIDIA for their contributions to this post. We also would like to thank Can Sun, Ophelia Yang, Aryaman Kothari, David Oh, and Ayush Kumar from AWS for their contributions.

    Amazon Dynamo Managed nvidia SageMaker
    Follow on Google News Follow on Flipboard
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email Copy Link
    myappsplus
    • Website

    Related Posts

    If Data Are Biased, AI Will Only Amplify Problems

    November 18, 2026

    Hacks, AI or overdue updates

    November 18, 2026

    AI workflow automation: 3 monday.com Elevate insights

    November 18, 2026
    Add A Comment
    Leave A Reply Cancel Reply

    Top Posts

    UiPath Launches AI Coding Agent, Integrates Claude Code and Codex to Target Development Automation

    November 16, 202642 Views

    The pros and cons of using your phone as a mobile hotspot

    November 16, 202640 Views

    Google’s new AI shopping cart tracks deals across stores

    October 25, 202631 Views
    Latest Reviews

    Modern laptops ditched disc drives — this $39 Dell DW316 USB-powered DVD-RW writer brings them back

    myappsplusSeptember 9, 2026

    Apple CEO John Ternus says the best AI device is still the iPhone

    myappsplusSeptember 9, 2026

    Best reveals from today’s Nintendo Direct: New Kirby, Metroid Ravenous, GameCube, more

    myappsplusSeptember 9, 2026
    Stay In Touch
    • Facebook
    • YouTube
    • TikTok
    • WhatsApp
    • Twitter
    • Instagram

    Subscribe to Updates

    Get the latest tech news from FooBar about tech, design and biz.

    Most Popular

    iPhone Duo is official — price, release date, specs, and everything you need to know

    September 9, 20260 Views

    Apple’s iPhone Duo foldable has the same IP68 protection as Pixel Fold

    September 9, 20260 Views

    Samsung, take note: iPhone Duo supports stylus input better than any Galaxy Fold

    September 9, 20260 Views
    Our Picks

    Whisker offers up to $200 Litter-Robot savings during Black Friday Sale starting from $564

    November 18, 2026

    Motorola Razr Fold finally gets its Android 17 update

    November 18, 2026

    Google Store’s Black Friday coupons are in the mail

    November 18, 2026

    Subscribe to Updates

    Subscribe to our newsletter and get the latest tech news, app updates, AI trends, smartphone reviews, and exclusive deals delivered straight to your inbox.

    Facebook X (Twitter) Instagram Pinterest
    • About Us
    • Get In Touch
    • Disclaimer
    • Privacy Policy
    • Terms & Conditions
    © 2026 MyAppsPlus. All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.