{"data":{"slug":"changyeyu-llm-rl-visualized","name":"LLM-RL-Visualized","tagline":"🌟100+ 原创 LLM / RL 原理图📚，《大模型算法》作者巨献！💥（100+ LLM/RL Algorithm Maps ）","github_url":"https://github.com/changyeyu/LLM-RL-Visualized","owner":"changyeyu","repo":"LLM-RL-Visualized","owner_avatar_url":"https://avatars.githubusercontent.com/u/202770431?v=4","primary_language":"Python","stars":4632,"forks":444,"topics":["ai","algorithm","deep-learning","llm","machine-learning","natural-language-processing","nlp-machine-learning","reinforcement-learning","transformers","vlm"],"archived":false,"github_pushed_at":"2026-07-06T11:52:39+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/changyeyu-llm-rl-visualized","markdown_url":"https://www.graphcanon.com/tools/changyeyu-llm-rl-visualized.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/changyeyu-llm-rl-visualized","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=changyeyu-llm-rl-visualized","description":"🌟100+ 原创 LLM / RL 原理图📚，《大模型算法》作者巨献！💥（100+  LLM/RL Algorithm Maps ）","homepage_url":"https://book.douban.com/subject/37331056/","license":"Other","open_issues":3,"watchers":20,"ai_summary":null,"readme_excerpt":"<p align=\"center\">\n  <img src=\"src/assets/banner/幻灯片2.SVG\" alt=\"图解大模型算法：LLM / RL / VLM 核心技术图谱\" />\n</p>\n\n<p align=\"center\">\n  <a href=\"./src/README_EN.md\">\n    <img\n      alt=\"English Version\"\n      src=\"https://img.shields.io/badge/English-Version-blue?style=for-the-badge\"\n      height=\"45\"\n    />\n  </a>\n  &nbsp;&nbsp;\n  <a href=\"./README.md\">\n    <img\n      alt=\"Chinese 中文版本\"\n      src=\"https://img.shields.io/badge/Chinese-中文版本-red?style=for-the-badge\"\n      height=\"45\"\n    />\n  </a>\n</p>\n\n---\n\n## 简 介\n\n🎉 **原创 100+ 架构图，系统讲解大模型、强化学习**，涵盖：LLM / VLM 等大模型原理、训练算法（RL、RLHF、GRPO、DPO、SFT 与 CoT 蒸馏等）、效果优化与 RAG 等。  \n\n🎉 关于架构图<strong>更详细</strong>的解读可参考：<a href=\"https://book.douban.com/subject/37331056/\">《大模型算法：强化学习、微调与对齐》</a> (豆瓣高分，多次京东AI图书Top 5 ！)\n\n🎉 本仓库**长期勘误、追加**，欢迎点击仓库顶部的 **Star ⭐** 关注，感谢鼓励✨\n\n🎉 单击图片可看高清大图，或浏览仓库目录中的 `.svg` 格式矢量图（活图，可无限缩放、可选择文字）\n\n<div align=\"center\">\n\n<a href=\"https://hellogithub.com/repository/changyeyu/LLM-RL-Visualized\" target=\"_blank\"><img src=\"https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=89dd575abef147188252e25675feb7aa&claim_uid=59EQqBGDeXApw2L&theme=small\" alt=\"Featured｜HelloGitHub\" /></a>\n\n</div>\n\n---\n\n\n## 目 录\n\n### 第1部分：大模型、强化学习的技术全景图\n- [大模型算法总体架构（以LLM、VLM为主）](#header-1)  \n- [强化学习算法图谱 (rl‑algo‑map).pdf——全网最大！](#header-1.5)\n- [LLM结构总图——全网最大！](#header-2)  \n- [策略梯度(Policy Gradient)-强化学习(PPO&GRPO等)之根基.pdf](#header-2.1)  \n\n### 第2部分：大模型基础\n- [LLM（Large Language Model）结构](#header-3)  \n- [LLM生成与解码（Decoding）过程](#header-4)  \n- [LLM输入层](#header-5)  \n- [LLM输出层](#header-6)  \n- [多模态模型结构（VLM、MLLM …）](#header-7)  \n- [LLM训练流程](#header-8)\n\n### 第3部分：SFT（有监督微调）\n- [微调（Fine‑Tuning）技术分类](#header-9)  \n- [LoRA（1 of 2）](#header-10)  \n- [LoRA（2 of 2）](#header-11)  \n- [Prefix‑Tuning](#header-12)  \n- [TokenID与词元的映射关系](#header-13)  \n- [SFT的Loss（交叉熵）](#header-14)  \n- [指令数据的来源](#header-15)  \n- [多个数据的拼接（Packing）](#header-16)\n\n### 第4部分：DPO（直接偏好优化）\n- [RLHF与DPO的训练架构对比](#header-17)  \n- [Prompt的收集](#header-18)  \n- [DPO（Direct Preference Optimization）](#header-19)  \n- [DPO训练全景图](#header-20)  \n- [β参数对DPO的影响](#header-21)  \n- [隐式奖励差异对参数更新幅度的影响](#header-22)\n\n### 第5部分：免训练的大模型优化技术\n- [CoT（Chain of Thought）与传统问答的对比](#header-23)  \n- [CoT、Self‑consistency CoT、ToT、GoT](#header-24)  \n- [穷举搜索（Exhaustive Search）](#header-25)  \n- [贪婪搜索（Greedy Search）](#header-26)  \n- [波束搜索（Beam Search）](#header-27)  \n- [多项式采样（Multinomial Sampling）](#header-28)  \n- [Top‑K采样（Top‑K Sampling）](#header-29)  \n- [Top‑P采样（Top‑P Sampling）](#header-30)  \n- [RAG（检索增强生成, Retrieval‑Augmented Generation）](#header-31)  \n- [功能调用（Function Calling）](#header-32)\n\n### 第6部分：强化学习（RL）基础\n- [强化学习(Reinforcement Learning, RL)的发展历程](#header-33)  \n- [三大机器学习范式](#header-34)  \n- [强化学习的基础架构](#header-35)  \n- [强化学习的运行轨迹](#header-36)  \n- [马尔可夫链 vs 马尔可夫决策过程（MDP）](#header-37)  \n- [探索与利用问题（Exploration and Exploitation）](#header-38)  \n- [Ɛ‑贪婪策略下使用动态的Ɛ值](#header-39)  \n- [强化学习训练范式的对比](#header-40)  \n- [强化学习算法分类](#header-41)  \n- [回报（累计奖励，Return）](#header-42)  \n- [反向迭代并计算回报 G](#header-43)  \n- [奖励（Reward）、回报（Return）、价值（Value）的关系](#header-44)  \n- [价值函数 Qπ 与 Vπ 的关系](#header-45)  \n- [蒙特卡洛（Monte Carlo，MC）法预估状态 St 的价值](#header-46)  \n- [TD 目标与 TD 误差的关系（TD target and TD error）](#header-47)  \n- [TD(0)、多步 TD 与蒙特卡洛的关系](#header-48)  \n- [蒙特卡洛方法与 TD 方法的特性](#header-49)  \n- [蒙特卡洛、TD、DP、穷举搜索的关系](#header-50)  \n- [两种输入输出结构的 DQN（Deep Q-Network）模型](#header-51)  \n- [DQN 的实际应用示例](#header-52)  \n- [DQN 的“高估”问题](#header-53)  \n- [基于价值 vs 策略（Value-Based vs Policy-Based）](#header-54)  \n- [策略梯度（Policy Gradient）](#header-55)  \n- [多智能体强化学习（MARL，Multi-agent reinforcement learning）](#header-56)  \n- [多智能体 DDPG](#header-57)  \n- [模仿学习（IL，Imitation Learning）](#header-58)  \n- [行为克隆（BC，Behavior Cloning）](#header-59)  \n- [逆向强化学习（IRL，Inverse RL）](#header-60)  \n- [有模型（Model-Based） vs 无模型（Model-Free）](#header-61)  \n- [封建等级强化学习（Feudal RL）](#header-62)  \n- [分布价值强化学习（Distributional RL）](#header-63)\n\n### 第7部分：策略优化架构及衍生算法\n- [Actor‑Critic 架构](#header-64)  \n- [引入基线与优势函数 A （Advantage）的作用](#header-65)  \n- [GAE（广义优势估计,Generalized Advantage Estimation）算法](#header-","github_created_at":"2025-04-26T14:30:47+00:00","created_at":"2026-07-11T10:42:16.5628+00:00","updated_at":"2026-07-11T10:42:24.781325+00:00","categories":[{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"}],"tags":[{"slug":"reinforcement-learning","name":"reinforcement-learning"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"llm","name":"llm"},{"slug":"ai","name":"ai"},{"slug":"algorithm","name":"algorithm"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"nlp-machine-learning","name":"nlp-machine-learning"},{"slug":"natural-language-processing","name":"natural-language-processing"}],"trust":{"provenance":{"is_fork":false,"github_id":973223133,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:42:17.171Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":4,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:42:17.865Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:42:17.590Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T10:42:17.590Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-07-11T10:42:17.590Z"}}}}