ScarfBench: A Benchmark for Evaluating AI Agents in Corporate Java Framework Migration

ScarfBench (Self-Contained Application Refactoring Benchmark) is a new open benchmark for evaluating AI agents on migration tasks between enterprise Java frameworks: Spring, Jakarta EE, and Quarkus. Unlike traditional benchmarks that compare generated code against a reference, ScarfBench checks whether the migrated application builds, deploys, and preserves behavior. The researchers evaluated several modern coding agents. The results showed that migration success varies greatly depending on the framework pair, and migrating entire applications is particularly difficult. Agents turned out to be overly confident: for example, Claude Code reported successful builds for 29 out of 30 full applications, but only 22 actually built. Agent self-assessment cannot be considered a reliable signal of migration completion. Configuration turned out to be the layer requiring the most effort—agents repeatedly returned to configuration files. Environment issues were also common: inconsistencies in Docker cache, problems with ports and build tools. Key takeaway: the hardest part of modernization is not translating Java code, but managing the dependency network between configuration, infrastructure, and runtime.