Model Routing: Simple? Until You Hit Reality

Routing between AI models is deceptively complex. In tests with 417 tasks on AppWorld Test Challenge using CodeAct agent, GPT-4.1 cost $155 ($0.37/task) while Claude Sonnet cost $79 ($0.19/task), despite GPT-4.1 having lower per-token pricing. The difference came from caching: Sonnet's lower cache-read pricing gave it a large advantage in agent workloads with reused context. Task difficulty is often invisible at routing time, and routers must balance cost, quality, latency, compliance, and reliability simultaneously. Latency depends on infrastructure factors like hardware and cache state, not just model size. The authors built a router that treats routing as an optimization problem, not classification, producing a cost-accuracy frontier. Configuration 1 achieved 84% accuracy for $93 and 83s, reducing cost by 21% and latency by 9% compared to running Opus alone, with only 4% accuracy drop. A standard difficulty-based router (teal diamond) had similar accuracy but higher cost. The optimization is lightweight: ~6 ms and 2 kB memory per task.