Evaluating AI Agents: A Production Template with Strands and AgentCore from AWS and Motorway
Amazon/AWS
Amazon Web Services
Anthropic
Motorway, a British online car marketplace, together with AWS, developed an AI agent evaluation pipeline for searching cars from dealers. The solution reduced incorrect results from 1 in 8 to 1 in 50 queries and cut problem identification time from several hours to a few minutes. At its core is a two-phase evaluation strategy using Strands Agents SDK and Amazon Bedrock AgentCore.
Motorway, a UK-based online car marketplace where up to 8,000 dealers bid on 2,500 cars daily, has partnered with AWS PACE to build an AI agent for finding cars in dealers' inventories. The agent uses eight tools, combining structured filtering across 89 attributes with vector search powered by LanceDB and Amazon Titan Text Embeddings V2. To ensure agent reliability, Motorway and AWS built an evaluation pipeline that reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from hours to minutes. The pipeline employs a two-phase strategy: testing during build with the strands-agents-evals library, and production monitoring with Amazon Bedrock AgentCore Evaluations. Evaluation covers three levels: tool usage (threshold >95%), reasoning logic (>85%), and output quality (>90%). To handle LLM nondeterminism, the pass^k metric is used—the probability of successfully completing k consecutive attempts. The pipeline includes five deployment stages with quality gates. A companion repository provides a deployable template adaptable to other agents.
Source: AWS ML blog —
original