AI at Scale: How to Manage Adoption, Metrics, and Team Transformation Among 500+ Developers

Yuri Katser, Product Owner at Data Office of the operational directorate, and Vyacheslav Guch, AI Product Manager, share insights from over a year of implementing agentic engineering in a bank with more than 500 developers, data engineers, analysts, and support engineers. They discovered that traditional learning does little to increase adoption, and that the number of users does not indicate business effect. They shifted from a single Daily Usage metric to four adoption metrics: AI penetration, Active users, W50+, and W80+, where the last two represent the share of employees using AI on more than 50% and 80% of working days in a month. The bank does not aim to automate everything but targets a Pareto-optimal state where 80% of tasks are handled by agents, yielding 80% of results. They found that writing code is no longer the main bottleneck; instead, the quality of specifications, context available to agents, integrations, and the team's ability to restructure work processes matter most. They introduced proxy metrics like the share of AI-assisted tasks and autonomously delivered tasks to track movement toward autonomous development, while still monitoring classic engineering metrics like Defect Rate, Cycle Time, Lead Time, and Throughput. They observed that roles are transforming: developers converge to end-to-end engineer or Product Engineer, and data roles to Data Product Engineer. In a pilot, a mini-team of two developers and a business expert completed an integration service in the same time as a five-person team, maintaining quality and increasing throughput, but Lead Time remained unchanged. They emphasize that technology preparation is crucial, outlining four pillars of agentic engineering, with context and integrations being the most extensive and fragmented layers. They tested the BMAD orchestrator, which revealed surprises, including that architecture designed around constraints often works better than trying to remove them.