A new research paper explores the fundamental link between the output distributions of Large Language Models (LLMs) and the corpora used during their pretraining. The study focuses on the Empirical Next-Token Distribution (ENTD), which serves as a theoretical benchmark for model performance.
The Role of ENTD
The ENTD is defined as the unrestricted global minimizer of the next-token cross-entropy loss, the standard objective function for LLM pretraining. By comparing an LLM's predictions to this empirical distribution, researchers can measure how effectively a model has "learned" the statistical properties of its training data.
- Scale and Fidelity: The study finds that as model scale and training compute increase, the agreement between the model's output and the ENTD also increases.
- The Long Tail: Despite high overall agreement, there remains a "long tail" of sequences where the model's predictions significantly deviate from the training data statistics.
Sources of Discrepancy
The researchers investigated several factors that might cause a model to diverge from the ENTD. These include the inherent limitations of the transformer architecture, specificities in the training procedure, and noise within the finite-sample estimates of the ENTD itself.
"We hope our findings will encourage more work on 'data-centric mechanistic interpretability,'" the authors note, suggesting a shift in how researchers understand AI behavior.
This approach aims to open the "black box" of AI by examining how behaviors emerge from training data rather than focusing solely on how they are encoded within the model's weights.
