Awesome Language Model Analysis

This paper list focuses on the theoretical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning behavior, generalization ability, and other properties of language models through formal/mathematical analysis -- proofs, provable guarantees, bounds, expressivity results, convergence analysis, and similar. Papers that also include supporting experiments still count; purely empirical/observational papers do not.

Scope of this list:

• Currently, this list focuses on transformer-based models.
• We collect papers with a genuine theoretical contribution, instead of purely empirical studies or papers that aim to improve the performance of language models without theoretically analyzing why.

Limitations of this list:

• This list is not exhaustive, and we may miss some very important papers.
• This list is not well-organized yet, and we may need to reorganize the list in the future.
• Some popular topics are not well-covered yet, such as mechanistic engineering, probing, and interpretability.

Statistics of This paper list:

• Total number of different papers: 664
• For more detailed statistics, please refer to the end of this page.

If you have any suggestions or want to contribute, please feel free to open an issue or a pull request.

For details on how to contribute, please refer to the contribution guidelines.

You can also share your thoughts and discuss with others in the Discussions.

[!NOTE]
For uncategorized version, please refer to here.

Table of Content

• Awesome Language Model Analysis
• Table of Content
  • Phenomena of Interest
    • In-Context Learning
    • Chain-of-Thought
    • Hallucination
    • Reversal Curse
    • Scaling Laws / Emergent Abilities / Grokking / etc.
    • Knowledge / Memory Mechanisms
    • Training Dynamics / Landscape / Optimization / Fine-tuning / etc.
    • Learning / Generalization / Reasoning / Weak to Strong Generalization
    • Other Phenomena / Discoveries
  • Representational Capacity
    • What Can Transformer Do? / Properties of Transformer
    • What Can Transformer Not Do? / Limitation of Transformer
  • Architectural Effectivity
    • Layer-normalization
    • Tokenization / Embedding
    • Linear Attention / State Space Models / Recurrent Language Models / etc.
  • Training Paradigms
  • Mechanistic Engineering / Probing / Interpretability
  • Miscellanea

Phenomena of Interest

^ back to top ^

Categories focusing on different phenomena, properties, and behaviors observed in large language models (LLMs) and transformer-based models.

In-Context Learning

^ back to top ^

Papers focusing on the theoretical and empirical analysis of in-context learning in large language models.

paper list (