Improving EDA Agent Reliability with NVIDIA NeMo Gym Evaluation
· Engineer's Notes · Cem Koyluoglu
NVIDIA NeMo Gym helped Siemens Fuse reduce agent turns by 66% and raise verification scores from 55% to 95% in standard‑cell characterization workflows.
What happened
Siemens Fuse employed the NVIDIA NeMo Gym open‑evaluation environment to diagnose and improve the reliability of its Fuse EDA AI Agent during Liberty file generation for standard‑cell characterization. The agent was tested on a benchmark derived from the open‑source GF180MCU PDK, which includes 229 standard cells across 36 PVT corners. For tuning, each rollout limited the flow to two cells across the same 36 corners. Approximately 25 rollouts were executed using Claude Opus 4.8 in Claude Code as the fixed model, with each run lasting about 1.5 to 2 hours.
The evaluation loop recorded thinking traces, tool actions, and intermediate messages, then applied a domain‑specific verifier that checks coverage, validity, relative validation, accuracy, and integrity of the final Liberty artifacts. The composite verification score, defined as the product of binary gates for the four metrics multiplied by the fraction of values within tolerance, rose from roughly 55% to about 95% after several optimization cycles. Concurrently, the number of agent turns dropped from 155 to 53 (a 66% reduction) and output tokens fell from approximately 81 K to 29 K (a 64% reduction). The improvements were achieved by refining the agent’s skills and Model Context Protocol (MCP) tool descriptions based on verifier diagnostics.
Why it matters in production
Evaluating agents against final engineering artifacts rather than isolated tool calls addresses a key reliability barrier: early‑stage errors can silently propagate through multistage chip‑design workflows, corrupting downstream results. By enforcing coverage, validity, relative validation, accuracy, and integrity checks, the verifier ensures that any failure aborts the reward, preventing agents from producing plausible but incorrect outputs.
The 66% reduction in agent turns and 64% reduction in token usage directly lower latency and compute cost, especially when large language models are billed per token. Shorter interaction sequences also simplify debugging and reduce the surface for ambiguous instructions. Raising the composite verification score to 95% improves confidence that generated Liberty files meet design specifications, decreasing the risk of costly re‑spins in silicon tape‑out.
Furthermore, the repeatable NeMo Gym environment enables systematic tracking of error propagation, facilitating root‑cause analysis and continuous improvement. This systematic approach is essential for production adoption of agentic AI in safety‑critical domains such as EDA, where undetected errors can have expensive downstream consequences.
Engineering takeaways
- Implement a domain‑specific verifier that enforces coverage, validity, relative validation, accuracy, and integrity before rewarding an agent’s output.
- Use a controlled evaluation loop (e.g., NeMo Gym) to run baseline rollouts, capture full interaction traces, and compare results under identical conditions.
- Refine skill definitions and MCP tool descriptions with explicit JSON schemas to reduce ambiguous instructions and improve tool selection.
- Monitor agent turns and token consumption as key efficiency metrics; aim for reductions that translate to lower latency and cost.
- Iterate through repeated rollouts, using verifier diagnostics to guide skill and interface adjustments until verification scores converge toward the target reliability threshold.
Related work on this site
- Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
- RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
- YouTube AI Summarizer — Open-source Chrome extension on the Chrome Web Store: AI summaries, key points, deep analysis, a two-host AI podcast (Gemini TTS) and transcript-grounded chat for any YouTube video, using Groq or Ollama Cloud with the user’s own key.
- Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.
This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.