The Hidden Grammar of Language Models: Three Discoveries About Text Generation Structure

The author conducted a two-year study of the internal structure of language models and found that text generation follows stable grammatical paths, models build an internal map of sentence direction, and uncertainty arises only at later stages of specific word selection.

OpenAI