Data-Agent Pilot: Brief Template, 50-Question Acceptance and Five Metrics to Shut It Down

The GlowByte BI team, building on FanRuan's materials about data agents, argues that buying a tool is easy, but establishing a path from a business task to a working scenario is hard. They outline six steps for a pilot: selecting a testable scenario based on frequency, error cost, data boundary, and existence of a reference standard; formalizing a brief with fields like metrics, sources, deviation thresholds, and acceptance criteria; preparing data strictly for the scenario, checking five questions about metric calculation, data source, access rights, update timing, and model understanding of business terms; conducting acceptance using 30-50 real questions from the past quarter, categorizing answers into accurate hits, plausible but incorrect, and refusals, requiring zero plausible incorrect answers for key metrics; placing human checkpoints based on error cost and reversibility, defining who can stop the scenario; and measuring operations after two months using five metrics: share of accepted answers, freed time, time to reaction, number of new exceptions per week, and share of refusals which should be noticeable but not growing. They emphasize the need for a named scenario owner from the business side, not IT or an external integrator. After a pilot, six artifacts remain, making the second scenario cheaper. The pilot is not a technology test but a test of the company's ability to set tasks and sustain the scenario.