Agent Substrate: Kubernetes‑Optimized Agent Runtime

· Engineer's Notes · Cem Koyluoglu

Agent Substrate multiplexes agent sessions over a small pool of warm Kubernetes Pods, checkpointing idle state and restoring it on demand.

What happened

Agent Substrate is an open‑source runtime that pairs with NVIDIA Dynamo and KAI Scheduler to coordinate agent execution, inference, and shared cluster resources. It multiplexes many agent sessions over a small pool of warm Kubernetes worker Pods, checkpointing idle sessions with full memory and filesystem state and restoring them onto any free worker. The runtime targets three workload types: coding assistants that alternate between LLM calls and repository operations, reinforcement learning post‑training with bursty CPU‑heavy environments, and short‑lived GPU sandboxes for research compilation and benchmarking. Upstream contributions include request parking in the router to queue turns when all workers are busy, per‑container resource limits for microVM sandboxes, and a vendor‑neutral design for pausing and restoring GPU state using Kubernetes Dynamic Resource Allocation. NVIDIA Dynamo coordinates inference across GPU workers while Agent Substrate manages agent environments, with Dynamo’s router using session identity, KV cache location, and tool‑call boundaries to keep related model calls on workers that already hold reusable context. Dynamo exposes inference signals such as projected time to first token and GPU‑pool saturation back to Agent Substrate, which uses them to decide whether to keep an actor resident, suspend it during long inference waits, or resume it as the model response nears completion. KAI Scheduler places the Substrate worker Pods alongside GPU trainers and inference servers, and the combined stack runs on the NVIDIA DSX AI Factory Platform with Substrate as an independent CNCF Sandbox project integrating with DSX OS components.

Why it matters in production

Kubernetes was designed for long‑lived services, not for the short, intermittent steps that agents perform. Spinning up a new Pod for each agent step incurs API server, scheduler, kubelet, image pull, and network setup overhead that can take several seconds, which is unacceptable for a step that may only need a few milliseconds. Maintaining a warm Pod per agent wastes compute resources because agents spend most of their time idle. Agent Substrate’s checkpointing and restoration reduce cold‑start latency to the time required to restore memory and filesystem state, eliminating the need to traverse the Kubernetes control plane for each step. This improves resource utilization, lowers operational cost, and reduces latency for interactive workloads. The integration with NVIDIA Dynamo allows the runtime to receive real‑time inference metrics, enabling it to suspend agents during long GPU waits and resume them when the GPU is ready, further improving GPU utilisation and reducing idle time. The per‑container resource limits and device‑pluggability features protect shared sandboxes from runaway containers and enable safe GPU sharing across multiple agents, enhancing security and stability.

Engineering takeaways

  • Implement a control plane that maps many agent actors onto a smaller pool of Kubernetes worker Pods, checkpointing idle actors and restoring them on demand.
  • Use inference signals such as projected time to first token and GPU‑pool saturation to decide when to keep an actor resident, suspend it, or resume it, thereby aligning agent scheduling with GPU availability.
  • Enforce per‑container CPU and memory limits within microVM sandboxes to prevent a single container from starving or killing its neighbors.
  • Design a vendor‑neutral GPU pause and restore mechanism that records device properties with each snapshot and restores onto a matching device, falling back to a cold start if none is available.
  • Integrate request parking in the router to queue turns when all workers are busy, ensuring that requests are retried until a worker becomes free and preventing premature failures.

Related work on this site

  • Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
  • RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
  • YouTube AI Summarizer — Open-source Chrome extension on the Chrome Web Store: AI summaries, key points, deep analysis, a two-host AI podcast (Gemini TTS) and transcript-grounded chat for any YouTube video, using Groq or Ollama Cloud with the user’s own key.
  • Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.

This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.

Source: https://developer.nvidia.com/blog/agent-substrate-advancing-kubernetes-for-agents-at-scale/

← All tech news