Where LLMs Store Facts: Moving the Eiffel Tower from Paris to Rome
OpenAI
Researchers tested the ROME method on GPT-2 XL to edit a specific fact in the model's weights: changing the location of the Eiffel Tower from Paris to Rome. The experiment confirmed that the edit works across multiple phrasings and that neighboring facts remain intact. This demonstrates how parametric knowledge can be targeted without full retraining.
In an experiment, the ROME (Rank-One Model Editing) method was applied to the GPT-2 XL model to change the fact that the Eiffel Tower is in Paris to Rome. The model, with 1.56 billion parameters, was chosen because its weights are openly accessible and it is supported by the original ROME implementation. Before editing, the model consistently preferred Paris across several prompts, with probabilities ranging from 2% to 49%. After one targeted edit, the model shifted its preference to Rome in all tested formulations, and other facts about the Eiffel Tower remained unchanged. The method modifies a matrix in the feed-forward network (FFN), which acts as associative memory for facts. The experiment also highlighted that facts are not stored as discrete neurons but are distributed across the network, and that ROME can adjust a specific association without full retraining. Finally, the original weights were restored, confirming the edit was reversible.
- Abbreviations
- LLM = Large Language Model — большая языковая модель
- RAG = Retrieval-Augmented Generation — генерация с поиском по внешним источникам
- FFN = Feed-Forward Network — сеть прямого распространения
Source: Habr — хаб ИИ —
original