Evaluating AI Agents: A production blueprint with Strands and AgentCore from AWS and Motorway

Motorway runs a daily auction where up to 8,000 dealers bid on up to 2,500 vehicles. Working with AWS, they built an AI agent that lets dealers search stock using natural language queries instead of manual filtering. The agent exposes eight tools combining structured filtering across 89 vehicle attributes with vector similarity search via LanceDB and Amazon Titan Text Embeddings V2. To ensure reliability, they created a two-phase evaluation strategy: build-time testing with strands-agents-evals and production monitoring with Amazon Bedrock AgentCore Evaluations. A three-layer framework assesses tool usage (over 95% threshold), reasoning (over 85%), and output quality (over 90%). A pass^k metric handles non-determinism: for customer-facing agents, consecutive trial success matters most. The pipeline reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from hours to minutes. A companion repository provides a deployable blueprint adaptable for other agents.