Rescale deploys AI triage agent on AWS using AI‑DLC methodology
· Engineer's Notes · Cem Koyluoglu
Rescale built an automated test‑failure triage agent in a five‑day AI‑DLC workshop, cutting deployment blocking time by 20% and improving classification accuracy to 68%.
What happened
Rescale created an AI‑driven triage agent to classify nightly end‑to‑end test failures and suggest remediation. The agent runs as a pod in the existing Argo Workflows CI/CD pipeline on Amazon EKS. It invokes Claude Opus 4.8 on Amazon Bedrock to label each failure as an infrastructure issue, code bug, flaky test, or environment problem, retrieves historical patterns from a knowledge base, queries pod logs via the Model Context Protocol (MCP) integrated with Amazon CloudWatch, and emits a structured remediation report in JSON and HTML. The development effort was organized as a five‑day, in‑person AI‑DLC workshop guided by AWS architects. Days 1–2 focused on vision, requirements, and user story definition; days 2–4 covered architecture, interface contracts, and implementation; day 5 built a PyTest‑based evaluation harness and demonstrated the agent to senior leadership. By the end of the workshop the pipeline was operational in staging, and the agent has been running in production since.
Why it matters in production
The manual triage workflow required engineers to move from test reports to logs, historical patterns, and source code to identify root causes, often delaying weekly production releases. Automating the first‑pass classification reduces the time that test failures block deployments. Production metrics show the agent’s classification agreement with human triage rose from 44% at initial deployment to 68% after iterative improvements, and the time blocking production deployments decreased by 20%. Because the agent runs within the existing EKS cluster, no additional runtime environment is required, limiting operational overhead and simplifying IAM and networking configurations. The use of Bedrock for inference and CloudWatch for log retrieval consolidates dependencies, reducing latency and potential points of failure. The evaluation harness provides a regression gate, ensuring that accuracy does not regress as the model or prompts evolve, which is critical for maintaining reliability in a continuous integration pipeline.
Engineering takeaways
- Leverage existing CI/CD infrastructure: Deploy AI agents as containers within the same Kubernetes cluster that hosts the build pipeline to avoid extra runtime management and to enable direct triggers.
- Structure development with AI‑DLC mob sessions: Conduct joint requirement clarification and design reviews in real time, allowing the AI to ask and receive answers to over 20 clarification questions, which improves alignment and reduces rework.
- Implement a lightweight evaluation harness: Use a PyTest‑based framework with live and stubbed modes to establish a classification‑agreement baseline and to gate regressions before production rollout.
- Prefer native cloud services for dependencies: Switch log retrieval from third‑party services to Amazon CloudWatch via MCP to simplify the dependency chain and lower latency.
- Iterate on prompt engineering and golden datasets: Track classification agreement (e.g., from 44% to 68%) and continuously refine prompts and training data to improve accuracy without changing the underlying model.
Related work on this site
- Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
- RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
- YouTube AI Summarizer — Open-source Chrome extension on the Chrome Web Store: AI summaries, key points, deep analysis, a two-host AI podcast (Gemini TTS) and transcript-grounded chat for any YouTube video, using Groq or Ollama Cloud with the user’s own key.
- Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.
This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.