What We Learned by Reproducing 2,200 ICML Papers

Hugging Face and alphaXiv organized the ICML 2026 Open Reproductions challenge from July 15 to August 2, 2026, inviting participants to reproduce papers from the conference using AI coding agents like Claude Code, Codex, Cursor, and Pi. Participants received $20 in compute credits and launched 2,962 cloud jobs; where full reproduction was impossible, toy reproductions on synthetic data were used. Of the 2,200 papers examined, 1,103 (51%) had at least one claim independently verified, including 266 fully reproduced and 632 partially reproduced with no falsifications; 3,978 individual claims were confirmed. However, 496 (23%) had at least one claim falsified or contested, including 49 with all claims falsified and 242 where independent teams reached opposite verdicts on the same claims. Notable confirmed falsifications include a paging paper's proof failing after k=1024, a theorem collapsing at step 224, a theory paper using reverse KL while code used forward KL, and an evaluation diluted by EOS padding tokens. The organizers re-verified all claimed falsifications and contacted authors, who responded positively. The challenge also highlighted that agents hit limits, such as local loops and misinterpretations, and that human oversight was essential for reliable results, as seen in the human-in-the-loop winner who personally judged image pairs. The event is considered the largest open, claim-by-claim audit of a machine learning conference, and all logbooks, verdicts, traces, and artifacts are public.