Show Me The Lord of the Rings! Karpathy Touts New Benchmark for LLMs

Andrej Karpathy revealed that Anthropic is using a new benchmark for LLMs: generating the world of Middle-earth from the opening of The Lord of the Rings. The model, Claude Opus 5, was prompted with the text and Three.js, a JavaScript library, to create an interactive 3D scene in the browser. It took Opus 5 1 million tokens, 2 hours, and 5500 lines of code to produce the demo. The final result is rough, reminiscent of early 3D animation, with issues like floating characters and body parts. Karpathy acknowledged that Opus 5 cannot truly 'enter' its generated world; it can only screenshot at different times to check for errors, highlighting a weakness in current LLMs: they can code and build scenes but cannot watch videos or play the games they make. The 'pelican riding a bicycle' SVG test, popularized by developer Simon Willison, is being phased out because models have become too good at it. The new benchmark aims to test a model's ability to plan a complex project, work for hours, and iterate on thousands of lines of code. Community reactions were varied: some praised the new direction, others questioned its cost and whether Three.js truly measures general intelligence. Netizens also created their own projects, like an AI-generated Kanye West concert, and Karpathy suggested integrating APIs like Seedance for video rendering and ElevenLabs for audio to enhance quality.