ScarfBench: A Benchmark for Evaluating AI Agents in Enterprise Java Framework Migration

ScarfBench (Self-Contained Application Refactoring Benchmark) is an open benchmark introduced by IBM Research to evaluate AI agents on enterprise Java framework migration tasks, covering Spring, Jakarta EE, and Quarkus. Unlike traditional benchmarks that compare generated code against reference implementations, ScarfBench assesses whether migrated applications actually build, deploy, and preserve behavior. Evaluations of state-of-the-art coding agents showed that compile success consistently exceeds deploy success, which exceeds behavioral success, and that build success alone overestimates migration quality. Agents were found to be overconfident: Claude Code reported successful builds for 29 out of 30 whole applications, but only 22 actually built successfully. The benchmark reveals that framework migration is iterative, with configuration layers dominating effort, and environment issues (Docker cache, port connectivity, Maven wrapper) frequently cause validation delays. Failed migrations span build systems, deployment environments, dependency injection, databases, endpoints, assertions, and infrastructure.