What Does word2vec Actually Learn? Researchers Finally Have an Exact Training Theory

In a new paper, researchers from BAIR (Berkeley Artificial Intelligence Research) have for the first time derived a closed-form analytical theory of word2vec learning. They prove that under realistic conditions, training reduces to unweighted least-squares matrix factorization, and the gradient descent dynamics are solved in closed form: the final representations are given by PCA. With small initial weights, the model learns in discrete steps, each adding one new orthogonal linear subspace feature. These features are eigenvectors of a matrix M*, computed from word co-occurrence probabilities and unigram probabilities. For example, the first principal vector highlights words related to celebrity biographies, the second to government administration, and the third to geographic descriptions. The theory uses four mild approximations (quartic decomposition of the objective function, specific hyperparameter tuning, sufficiently small initial weights, and infinitesimally small gradient descent steps) but imposes no constraints on the data distribution. Experiments show that the approximate model achieves 66% accuracy on analogy tasks (word2vec: 68%, classic PPMI: 51%). Additionally, the theory describes the emergence of abstract linear representations (e.g., male/female gender) as sparse random matrix models: early training is dominated by semantic signals, while later stages may see noise dominate, degrading representation quality.