Cost‑Optimizing GPU‑Based Speech‑to‑Text Inference with EC2 Spot
· Engineer's Notes · Cem Koyluoglu
AssemblyAI reduces inference costs by 70 % using EC2 Spot GPU instances, queue‑based retries, and multi‑family diversification.
What happened
AssemblyAI runs speech‑to‑text inference on NVIDIA GPUs. Each audio file triggers a pipeline of acoustic, language, and post‑processing models. Thousands of GPU instances run continuously to meet latency requirements. On‑Demand pricing would be prohibitive, so the company adopted EC2 Spot GPU instances. Spot instances can be reclaimed with a 2‑minute warning. To maintain availability and latency guarantees, AssemblyAI built an architecture that absorbs Spot interruptions while keeping data within specific geographic boundaries for regulated customers. The solution combines Amazon ECS, EC2 Spot, and Amazon SQS. ECS capacity providers mix Spot and On‑Demand instances across g4dn, g5, and g6 GPU families. A queue‑based retry mechanism uses SQS to recover interrupted jobs. The platform runs in US West (Oregon) and Europe (London) regions, each with independent queues, clusters, and capacity providers, satisfying data residency requirements. The result is a 70 % cost reduction on inference compute compared to On‑Demand pricing, even as compute demand grew nearly four times year over year.
Why it matters in production
The architecture demonstrates how to balance cost, reliability, and compliance for GPU‑intensive inference. Spot instances provide significant savings but introduce interruption risk. By diversifying across three GPU families—g4dn (NVIDIA T4), g5 (NVIDIA A10G), and g6 (NVIDIA L4)—the fleet taps independent capacity pools, reducing the likelihood of simultaneous shortages. Queue‑based retries decouple job submission from execution; when a Spot instance receives an interruption notice, the in‑flight job is returned to the main SQS queue, allowing another healthy task to pick it up. This design limits recovery time to the 2‑minute interruption window and keeps SLA commitments intact. Fast AMI warm‑up (≈2 minutes) further reduces latency for new workers. CloudWatch metrics provide observability of fleet health and interruption events. Regional deployment ensures that audio data never leaves the required geographic boundaries, meeting regulatory constraints in healthcare, finance, and legal sectors. The 70 % cost saving directly improves unit economics, enabling the platform to scale sustainably as demand quadruples.
Engineering takeaways
- Diversify Spot capacity across multiple GPU families and Availability Zones to increase aggregate compute availability.
- Use ECS capacity providers with a weighted strategy that includes an On‑Demand baseline and a Spot pool for demand spikes.
- Decouple job submission from execution with SQS; implement a Spot interruption handler that returns unfinished jobs to the queue.
- Create lightweight AMIs containing model artifacts to enable rapid worker startup (≈2 minutes).
- Deploy independent clusters and queues per region to satisfy data residency requirements and isolate failures.
- Monitor fleet health and interruption events with CloudWatch metrics to maintain observability and trigger automated scaling.
Related work on this site
- RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
- Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
- Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.
This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.