Unbiased Evaluation Unlocks Cost‑Effective Vision Models on Bedrock
· Engineer's Notes · Cem Koyluoglu
A real‑estate platform discovered evaluation bias, rebuilt ground truth, and achieved 86% lower cost with a Bedrock model.
What happened
A digital real‑estate application processed tens of thousands of property images daily, evaluating 11 quality attributes such as watermark detection, room classification, furnishing status, photo quality, and construction indicators. The evaluation dataset originally contained 96.8 percent of labels inherited from a single production model’s predictions, with only 3.2 percent verified by human experts. Hidden errors—such as character‑level mistakes in watermark extraction and forced single labels on multi‑room images—further skewed the dataset. When the team migrated the vision workload to Amazon Bedrock, the biased labels caused every alternative model to appear underperforming.
To break the circular evaluation, the team hand‑annotated 356 images from the UAE and 230 from Egypt. The first 256 UAE images formed a development set; the remaining 100 UAE images were added to the evaluation set. A strategy and analytics expert performed the annotations, and a dedicated clarification team resolved ambiguous cases, ensuring consistency across the dataset.
The switch from single‑label to multi‑label classification addressed artificial ambiguity, allowing a hallway photo showing a bedroom through a doorway to receive both labels. The team also introduced region‑specific rules to filter decorative Arabic text overlays that were incorrectly treated as watermarks.
Why it matters in production
Bias in evaluation data creates a circular dependency: models are judged against the behavior of the incumbent model rather than against true correctness. This leads to inflated error rates for new models, discourages experimentation, and can mask real performance gains. In production, such bias can result in higher costs, degraded user experience, and missed opportunities for cost‑effective model upgrades.
Rebuilding ground truth with independent human verification restores the validity of model comparisons. The unbiased dataset enabled the team to apply evolutionary prompt optimization, treating each prompt version as an agent in a reinforcement‑learning‑style loop. Micro F1 was used as the reward signal, while error analysis guided prompt mutations. This process produced a final prompt (v27_PF) that achieved a Micro F1 of 0.80 on the 356‑image evaluation set, compared to 0.56 for the legacy model. The relative gain of 47 percent demonstrates the impact of unbiased evaluation.
Cost savings were also realized: the new Bedrock model delivered 86 percent lower cost while maintaining or exceeding accuracy across all 11 attributes. The reduction in evaluation bias directly translated to a more reliable cost‑benefit assessment, enabling the platform to adopt a more economical solution without compromising quality.
Engineering takeaways
- Validate evaluation datasets with fresh, independent human annotations before model comparison.
- Convert single‑label tasks to multi‑label where domain data naturally contains overlapping categories.
- Use domain‑specific rules (e.g., regional text overlays) to reduce false positives.
- Employ a reinforcement‑learning‑style prompt optimization loop, using Micro F1 as the reward and error analysis as the gradient.
- Measure both Micro F1 and Macro F1 to detect performance gaps on rare classes.
Related work on this site
- RAG & Grounded LLM Systems — LLM features that answer from the right source instead of from memory: document-grounded assistants, transcript-grounded chat and quality gates around generated output.
- Agentic & LLM Automation — Multi-step LLM pipelines that run unattended: model cascades with fallbacks, validation gates, scheduled automation and alerting when something breaks.
- Automated AI News Pipeline (this site) — The pipeline behind this site’s Tech News: scheduled GitHub Actions scrape sources, an LLM cascade on Groq rewrites and enhances articles, and layered quality gates (date integrity, language checks, instruction-leak detection, duplicate detection) decide what gets published, with Telegram alerts.
This note was drafted with AI assistance from the primary source credited on this page, and automatically checked against that source before publishing.