Airbnb uses cost‑per‑task metrics to manage coding agents on Amazon Bedrock

· Engineer's Notes · Cem Koyluoglu

Airbnb evaluates and monitors coding agents on Bedrock using cost‑per‑task, latency, and throttling metrics to guide model selection and capacity planning.

What happened

Airbnb runs thousands of weekly coding requests through an internal tool called AirChat, which forwards prompts to Amazon Bedrock via a custom LLM gateway. Over a five‑week period eight new foundation models were released, and by mid‑October roughly two‑thirds of the requests were routed to those new models. Before each rollout Airbnb runs side‑by‑side evaluations on a fixed suite of engineering tasks, measuring both a quality score and the list‑price cost of all tokens required to complete a task (cost per task). After deployment, CloudWatch metrics—InvocationLatency, TimeToFirstToken, OutputTokenCount, InvocationThrottles, EstimatedTPMQuotaUsage, and token‑type counts—are used to compare real‑world behavior against the pre‑release evals. The team tracks latency spikes, throttling events, and changes in token consumption, adjusting model defaults and effort ceilings (the maximum reasoning effort allowed) when discrepancies appear. One model, referred to as model A, showed a three‑point quality gain in evals with comparable cost per task, leading to its rollout with high effort capped and lower effort encouraged for most work.

Why it matters in production

The approach highlights several production concerns. First, cost per task provides a concrete financial anchor for model selection, allowing teams to balance quality gains against token‑price impacts; a 0.8‑point score increase at high reasoning effort incurred a 2.5× cost rise in a test. Second, latency and throughput metrics (TimeToFirstToken, OTPS) expose performance regressions that raw model benchmarks miss; a 40 % latency increase was traced to a prompt change that also raised output tokens by 50 %. Third, throttling signals (InvocationThrottles, EstimatedTPMQuotaUsage) enable proactive quota management, preventing rate‑limit interruptions by setting alarms before token‑per‑minute limits are reached. Fourth, the side‑by‑side eval framework reduces risk when new models arrive, as production data can be quickly reconciled with expected cost per task, revealing mismatches such as a model that performed well in evals but exhibited higher real‑world spend. Finally, aggregating usage by model, team, and user supports capacity planning and cost allocation, ensuring that engineering‑adjacent users do not unintentionally overload shared resources.

Engineering takeaways

  • Instrument every Bedrock request with token‑type counts and latency metrics; derive OTPS to monitor generation speed.
  • Run automated evals on a representative task suite before each model rollout, recording both quality scores and cost per task using public list prices.
  • Compare post‑deployment CloudWatch signals against eval baselines; trigger alerts on latency, token‑count, or throttling deviations.
  • Enforce effort ceilings in the gateway to cap cost escalation when higher reasoning effort is not justified.
  • Maintain a centralized table of model list‑price timestamps to ensure consistent cost calculations across eval and production data.

Related work on this site

  • RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
  • Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
  • Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.

This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.

Source: https://aws.amazon.com/blogs/machine-learning/how-airbnb-uses-cost-per-task-to-operate-coding-agents-on-amazon-bedrock/

← All tech news