Efficient Agentic Shopping Assistant via Sequence-Level Knowledge Distillation on SageMaker AI
· Engineer's Notes · Cem Koyluoglu
How to reduce cost and latency of multi-step shopping agents by distilling a frontier model into a smaller student using SeqKD on Amazon SageMaker AI.
What happened
The blog post describes a workflow that reduces the per‑task cost of a shopping‑assistant agent by applying sequence‑level knowledge distillation (SeqKD). A frontier teacher model (GLM‑5.3 on Amazon Bedrock) generates multi‑step reasoning and tool‑call trajectories for the ShoppingBench benchmark, which contains 750 synthesized shopping problems (250 each for product, shop, and voucher intents). The problems are split into 426 training instances and 324 held‑out evaluation instances (product 103, shop 105, voucher 116). For each training problem the teacher is sampled twice, producing 850 raw trajectories. A programmatic scorer keeps only the trajectories that solve the task, resulting in roughly 2,600 training rows and 130 validation rows after step‑wise encoding. The trajectories are encoded as chat‑style messages, each row containing the conversation so far (user) and the next reasoning‑and‑action step (assistant). The student model (Qwen3.5‑4B) is fine‑tuned serverlessly using SageMaker AI model customization, then evaluated on the held‑out split and deployed behind a real‑time SageMaker AI inference endpoint. Prerequisites include an AWS account with IAM permissions, access to Amazon Bedrock, a SageMaker Studio notebook (minimum ml.m5.4xlarge instance), Python 3.12+, SageMaker SDK v3, and boto3.
Why it matters in production
This approach directly addresses the high per‑call cost of agentic applications, where a single user request can trigger multiple frontier‑model invocations (search, read, refine, verify, decide). By replacing the expensive teacher with a distilled student, latency improves because the student model is smaller, and operational cost drops as the student can be hosted on cheaper inference infrastructure. The reliance on only text outputs (no logits or gradients) makes SeqKD viable for closed or API‑only teachers, preserving the ability to leverage powerful frontier models without incurring their runtime price. The use of serverless fine‑tuning removes the need for dedicated training clusters, further lowering cost and simplifying scaling. However, the pipeline introduces additional steps—trajectory generation, scoring, and row‑wise encoding—that add upfront compute overhead. Validation must be problem‑level to avoid data leakage, and token‑length limits require careful row construction to avoid truncation of critical context. Security considerations include protecting the IAM role that can invoke the teacher model and ensuring that generated trajectories do not expose sensitive catalog data.
Engineering takeaways
- Use SeqKD when the teacher model is accessible only via API; it requires only the teacher’s text outputs and avoids logit‑based or on‑policy methods.
- Apply rejection sampling with a programmatic scorer to filter teacher trajectories; this converts an imperfect teacher into a high‑quality training set.
- Encode each trajectory step as a separate chat turn and split data by problem, not by row, to prevent validation leakage.
- Leverage SageMaker AI serverless fine‑tuning to eliminate dedicated training infrastructure and to scale cost‑effectively.
- Deploy the distilled student behind a SageMaker AI real‑time endpoint to achieve lower latency and reduced inference cost compared to the frontier teacher.
Related work on this site
- Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
- RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
- YouTube AI Summarizer — Open-source Chrome extension on the Chrome Web Store: AI summaries, key points, deep analysis, a two-host AI podcast (Gemini TTS) and transcript-grounded chat for any YouTube video, using Groq or Ollama Cloud with the user’s own key.
- Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.
This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.