---
title: "LLM-RL-Visualized"
type: "tool"
slug: "changyeyu-llm-rl-visualized"
canonical_url: "https://www.graphcanon.com/tools/changyeyu-llm-rl-visualized"
github_url: "https://github.com/changyeyu/LLM-RL-Visualized"
homepage_url: "https://book.douban.com/subject/37331056/"
stars: 4632
forks: 444
primary_language: "Python"
license: "Other"
archived: false
categories: ["ai-agents", "vector-databases", "llm-frameworks"]
tags: ["reinforcement-learning", "deep-learning", "llm", "ai", "algorithm", "machine-learning", "nlp-machine-learning", "natural-language-processing"]
updated_at: "2026-07-11T10:42:24.781325+00:00"
---

# LLM-RL-Visualized

> 🌟100+ 原创 LLM / RL 原理图📚，《大模型算法》作者巨献！💥（100+ LLM/RL Algorithm Maps ）

🌟100+ 原创 LLM / RL 原理图📚，《大模型算法》作者巨献！💥（100+ LLM/RL Algorithm Maps ）

## Facts

- Repository: https://github.com/changyeyu/LLM-RL-Visualized
- Homepage: https://book.douban.com/subject/37331056/
- Stars: 4,632 · Forks: 444 · Open issues: 3 · Watchers: 20
- Primary language: Python
- License: Other
- Last pushed: 2026-07-06T11:52:39+00:00

## Trust & health

_Signals computed from public GitHub metadata. Not a security guarantee._

- Maintenance: Very active (computed 2026-07-11T10:42:17.171Z)
- Security scan: No lockfile (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-11T10:42:17.865Z
- Full report: [trust report](/tools/changyeyu-llm-rl-visualized/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/changyeyu-llm-rl-visualized/trust)

## Categories

- [AI Agents](/categories/ai-agents.md)
- [Vector Databases](/categories/vector-databases.md)
- [LLM Frameworks](/categories/llm-frameworks.md)

## Tags

reinforcement-learning, deep-learning, llm, ai, algorithm, machine-learning, nlp-machine-learning, natural-language-processing

## Category neighbours (exploratory)

_Same-category tools for discovery only - not curated alternatives. Cap shown at six._

- [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active]
- [ECC](/tools/affaan-m-ecc.md) - The agent harness performance optimization system for AI agents (★ 228,395) [Very active]
- [hermes-agent](/tools/nousresearch-hermes-agent.md) - The agent that grows with you (★ 212,994) [Very active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active]

_+ 2 more not listed._

## README (excerpt)

_Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._

```text
<p align="center">
  <img src="src/assets/banner/幻灯片2.SVG" alt="图解大模型算法：LLM / RL / VLM 核心技术图谱" />
</p>

<p align="center">
  <a href="./src/README_EN.md">
    <img
      alt="English Version"
      src="https://img.shields.io/badge/English-Version-blue?style=for-the-badge"
      height="45"
    />
  </a>
  &nbsp;&nbsp;
  <a href="./README.md">
    <img
      alt="Chinese 中文版本"
      src="https://img.shields.io/badge/Chinese-中文版本-red?style=for-the-badge"
      height="45"
    />
  </a>
</p>

---

## 简 介

🎉 **原创 100+ 架构图，系统讲解大模型、强化学习**，涵盖：LLM / VLM 等大模型原理、训练算法（RL、RLHF、GRPO、DPO、SFT 与 CoT 蒸馏等）、效果优化与 RAG 等。  

🎉 关于架构图<strong>更详细</strong>的解读可参考：<a href="https://book.douban.com/subject/37331056/">《大模型算法：强化学习、微调与对齐》</a> (豆瓣高分，多次京东AI图书Top 5 ！)

🎉 本仓库**长期勘误、追加**，欢迎点击仓库顶部的 **Star ⭐** 关注，感谢鼓励✨

🎉 单击图片可看高清大图，或浏览仓库目录中的 `.svg` 格式矢量图（活图，可无限缩放、可选择文字）

<div align="center">

<a href="https://hellogithub.com/repository/changyeyu/LLM-RL-Visualized" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=89dd575abef147188252e25675feb7aa&claim_uid=59EQqBGDeXApw2L&theme=small" alt="Featured｜HelloGitHub" /></a>

</div>

---


## 目 录

### 第1部分：大模型、强化学习的技术全景图
- [大模型算法总体架构（以LLM、VLM为主）](#header-1)  
- [强化学习算法图谱 (rl‑algo‑map).pdf——全网最大！](#header-1.5)
- [LLM结构总图——全网最大！](#header-2)  
- [策略梯度(Policy Gradient)-强化学习(PPO&GRPO等)之根基.pdf](#header-2.1)  

### 第2部分：大模型基础
- [LLM（Large Language Model）结构](#header-3)  
- [LLM生成与解码（Decoding）过程](#header-4)  
- [LLM输入层](#header-5)  
- [LLM输出层](#header-6)  
- [多模态模型结构（VLM、MLLM …）](#header-7)  
- [LLM训练流程](#header-8)

### 第3部分：SFT（有监督微调）
- [微调（Fine‑Tuning）技术分类](#header-9)  
- [LoRA（1 of 2）](#header-10)  
- [LoRA（2 of 2）](#header-11)  
- [Prefix‑Tuning](#header-12)  
- [TokenID与词元的映射关系](#header-13)  
- [SFT的Loss（交叉熵）](#header-14)  
- [指令数据的来源](#header-15)  
- [多个数据的拼接（Packing）](#header-16)

### 第4部分：DPO（直接偏好优化）
- [RLHF与DPO的训练架构对比](#header-17)  
- [Prompt的收集](#header-18)  
- [DPO（Direct Preference Optimization）](#header-19)  
- [DPO训练全景图](#header-20)  
- [β参数对DPO的影响](#header-21)  
- [隐式奖励差异对参数更新幅度的影响](#header-22)

### 第5部分：免训练的大模型优化技术
- [CoT（Chain of Thought）与传统问答的对比](#header-23)  
- [CoT、Self‑consistency CoT、ToT、GoT](#header-24)  
- [穷举搜索（Exhaustive Search）](#header-25)  
- [贪婪搜索（Greedy Search）](#header-26)  
- [波束搜索（Beam Search）](#header-27)  
- [多项式采样（Multinomial Sampling）](#header-28)  
- [Top‑K采样（Top‑K Sampling）](#header-29)  
- [Top‑P采样（Top‑P Sampling）](#header-30)  
- [RAG（检索增强生成, Retrieval‑Augmented Generation）](#header-31)  
- [功能调用（Function Calling）](#header-32)

### 第6部分：强化学习（RL）基础
- [强化学习(Reinforcement Learning, RL)的发展历程](#header-33)  
- [三大机器学习范式](#header-34)  
- [强化学习的基础架构](#header-35)  
- [强化学习的运行轨迹](#header-36)  
- [马尔可夫链 vs 马尔可夫决策过程（MDP）](#header-37)  
- [探索与利用问题（Exploration and Exploitation）](#header-38)  
- [Ɛ‑贪婪策略下使用动态的Ɛ值](#header-39)  
- [强化学习训练范式的对比](#header-40)  
- [强化学习算法分类](#header-41)  
- [回报（累计奖励，Return）](#header-42)  
- [反向迭代并计算回报 G](#header-43)  
- [奖励（Reward）、回报（Return）、价值（Value）的关系](#header-44)  
- [价值函数 Qπ 与 Vπ 的关系](#header-45)  
- [蒙特卡洛（Monte Carlo，MC）法预估状态 St 的价值](#header-46)  
- [TD 目标与 TD 误差的关系（TD target and TD error）](#header-47)  
- [TD(0)、多步 TD 与蒙特卡洛的关系](#header-48)  
- [蒙特卡洛方法与 TD 方法的特性](#header-49)  
- [蒙特卡洛、TD、DP、穷举搜索的关系](#header-50)  
- [两种输入输出结构的 DQN（Deep Q-Network）模型](#header-51)  
- [DQN 的实际应用示例](#header-52)  
- [DQN 的“高估”问题](#header-53)  
- [基于价值 vs 策略（Value-Based vs Policy-Based）](#header-54)  
- [策略梯度（Policy Gradient）](#header-55)  
- [多智能体强化学习（MARL，Multi-agent reinforcement learning）](#header-56)  
- [多智能体 DDPG](#header-57)  
- [模仿学习（IL，Imitation Learning）](#header-58)  
- [行为克隆（BC，Behavior Cloning）](#header-59)  
- [逆向强化学习（IRL，Inverse RL）](#header-60)  
- [有模型（Model-Based） vs 无模型（Model-Free）](#header-61)  
- [封建等级强化学习（Feudal RL）](#header-62)  
- [分布价值强化学习（Distributional RL）](#header-63)

### 第7部分：策略优化架构及衍生算法
- [Actor‑Critic 架构](#header-64)  
- [引入基线与优势函数 A （Advantage）的作用](#header-65)  
- [GAE（广义优势估计,Generalized Advantage Estimation）算法](#header-
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/changyeyu-llm-rl-visualized`](/api/graphcanon/tools/changyeyu-llm-rl-visualized)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_