{"data":{"slug":"furyton-awesome-language-model-analysis","name":"awesome-language-model-analysis","tagline":"A curated list of papers focusing on the theoretical analysis of large language models.","github_url":"https://github.com/Furyton/awesome-language-model-analysis","owner":"Furyton","repo":"awesome-language-model-analysis","owner_avatar_url":"https://avatars.githubusercontent.com/u/26501227?v=4","primary_language":"Python","stars":101,"forks":1,"topics":["ai","analysis","analytics","awesome","chatgpt","deep-learning","generative-ai","large-language-models","llm","nlp","theory","transformers"],"archived":false,"github_pushed_at":"2026-07-08T15:52:06+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/furyton-awesome-language-model-analysis","markdown_url":"https://www.graphcanon.com/tools/furyton-awesome-language-model-analysis.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/furyton-awesome-language-model-analysis","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=furyton-awesome-language-model-analysis","description":"This paper list focuses on the theoretical and empirical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning behavior, generalization ability, and other properties of language models through theoretical analysis, empirical analysis, or a combination of both.","homepage_url":null,"license":"CC0-1.0","open_issues":0,"watchers":3,"ai_summary":"This repository contains a collection of 664 papers that focus on the theoretical and empirical analysis of transformer-based language models, emphasizing their properties such as learning behavior and generalization ability through formal/mathematical proofs, provable guarantees, bounds, expressivity results, convergence analysis. The list excludes purely empirical studies.","readme_excerpt":"# Awesome Language Model Analysis \n\nThis paper list focuses on the **theoretical analysis** of language models, especially **large language models** (LLMs).\nThe papers in this list investigate the learning behavior, generalization ability, and other properties of language models through formal/mathematical analysis -- proofs, provable guarantees, bounds, expressivity results, convergence analysis, and similar. Papers that also include supporting experiments still count; purely empirical/observational papers do not.\n\nScope of this list:\n- Currently, this list focuses on **transformer-based** models.\n- We collect papers with a genuine theoretical contribution, instead of purely empirical studies or papers that aim to improve the performance of language models without theoretically analyzing why.\n\nLimitations of this list:\n- This list is not exhaustive, and we may miss some very important papers.\n- This list is not well-organized yet, and we may need to reorganize the list in the future.\n- Some popular topics are not well-covered yet, such as mechanistic engineering, probing, and interpretability.\n\nStatistics of This paper list:\n- Total number of different papers: **664**\n- For more detailed statistics, please refer to the end of this page.\n\nIf you have any suggestions or want to contribute, please feel free to open an issue or a pull request.\n\nFor details on how to contribute, please refer to the [contribution guidelines](CONTRIBUTING.md).\n\nYou can also share your thoughts and discuss with others in the [Discussions](https://github.com/Furyton/awesome-language-model-analysis/discussions).\n\n> [!NOTE]  \n> For uncategorized version, please refer to [here](README.uncategorized.md).\n\nTable of Content\n====================\n\n- [Awesome Language Model Analysis](#awesome-language-model-analysis-)\n- [Table of Content](#table-of-content)\n  - [**Phenomena of Interest**](#phenomena-of-interest)\n    - [**In-Context Learning**](#in-context-learning)\n    - [**Chain-of-Thought**](#chain-of-thought)\n    - [**Hallucination**](#hallucination)\n    - [**Reversal Curse**](#reversal-curse)\n    - [**Scaling Laws / Emergent Abilities / Grokking / etc.**](#scaling-laws--emergent-abilities--grokking--etc)\n    - [**Knowledge / Memory Mechanisms**](#knowledge--memory-mechanisms)\n    - [**Training Dynamics / Landscape / Optimization / Fine-tuning / etc.**](#training-dynamics--landscape--optimization--fine-tuning--etc)\n    - [**Learning / Generalization / Reasoning / Weak to Strong Generalization**](#learning--generalization--reasoning--weak-to-strong-generalization)\n    - [**Other Phenomena / Discoveries**](#other-phenomena--discoveries)\n  - [**Representational Capacity**](#representational-capacity)\n    - [**What Can Transformer Do? / Properties of Transformer**](#what-can-transformer-do--properties-of-transformer)\n    - [**What Can Transformer Not Do? / Limitation of Transformer**](#what-can-transformer-not-do--limitation-of-transformer)\n  - [**Architectural Effectivity**](#architectural-effectivity)\n    - [**Layer-normalization**](#layer-normalization)\n    - [**Tokenization / Embedding**](#tokenization--embedding)\n    - [**Linear Attention / State Space Models / Recurrent Language Models / etc.**](#linear-attention--state-space-models--recurrent-language-models--etc)\n  - [**Training Paradigms**](#training-paradigms)\n  - [**Mechanistic Engineering / Probing / Interpretability**](#mechanistic-engineering--probing--interpretability)\n  - [**Miscellanea**](#miscellanea)\n\n\n\n## **Phenomena of Interest**\n\n**[`^        back to top        ^`](#awesome-language-model-analysis-)**\n\nCategories focusing on different phenomena, properties, and behaviors observed in large language models (LLMs) and transformer-based models.\n\n### **In-Context Learning**\n\n**[`^        back to top        ^`](#awesome-language-model-analysis-)**\n\nPapers focusing on the theoretical and empirical analysis of in-context learning in large language models.\n\n\n<details open>\n<summary><em>paper list (","github_created_at":"2024-04-25T14:55:57+00:00","created_at":"2026-07-11T10:33:33.878809+00:00","updated_at":"2026-07-11T11:32:44.605548+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"awesome","name":"awesome"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"ai","name":"ai"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"generative-ai","name":"generative-ai"},{"slug":"chatgpt","name":"chatgpt"},{"slug":"analytics","name":"analytics"},{"slug":"analysis","name":"analysis"}],"trust":{"provenance":{"is_fork":false,"github_id":791883093,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:33:34.614Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":2,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":5,"high_count":0,"last_scan_at":"2026-07-11T10:33:35.594Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T11:32:14.095Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T11:32:14.095Z"},"license_spdx":{"value":"CC0-1.0","source":"github.license","observed_at":"2026-07-11T11:32:14.095Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Some knowledge in theoretical computer science or mathematics is advised to fully comprehend the papers listed.","Python proficiency might be beneficial for implementing models based on theoretical findings."]},"constraints":null,"when_to_use":["When you seek an in-depth theoretical understanding and formal/mathematical proofs related to the learning behavior and generalization ability of transformer-based large language models.","If you are interested in research involving formal proofs, provable guarantees, bounds, expressivity results, convergence analysis specifically for transformer-based models."],"when_not_to_use":["Avoid relying on this list if purely empirical or observational studies are more relevant to your needs as they are excluded from the repository.","You should not use this resource if a comprehensive coverage of mechanistic engineering, probing, and interpretability is required, as these topics are currently less covered."],"source":"enrich:decision_facts","observed_at":"2026-07-11T11:32:44.354Z"},"constraint_facets":null,"decision_summary":[{"label":"Requirements","value":"Some knowledge in theoretical computer science or mathematics is advised to fully comprehend the papers listed.; Python proficiency might be beneficial for implementing models based on theoretical findings."},{"label":"Adopt for","value":"Curated List of Theoretical Papers on Large Language Models"}]}}