{"data":{"slug":"harleyszhang-llm-note","name":"llm_note","tagline":"LLM notes covering model inference transformer structures and framework analysis","github_url":"https://github.com/harleyszhang/llm_note","owner":"harleyszhang","repo":"llm_note","owner_avatar_url":"https://avatars.githubusercontent.com/u/37138671?v=4","primary_language":"Python","stars":888,"forks":90,"topics":["cuda-programming","kv-cache","llm","llm-inference","transformer-models","triton-kernels","vllm"],"archived":false,"github_pushed_at":"2026-08-19T06:46:41+00:00","maintenance_label":"Very active","stars_delta_30d":-1,"url":"https://www.graphcanon.com/tools/harleyszhang-llm-note","markdown_url":"https://www.graphcanon.com/tools/harleyszhang-llm-note.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/harleyszhang-llm-note","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=harleyszhang-llm-note","description":"LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.","homepage_url":null,"license":null,"open_issues":0,"watchers":9,"ai_summary":"A collection of detailed notes from Harley Szhang on aspects related to large language models including their structure, inference methods, and an in-depth look at LLM frameworks.","readme_excerpt":"- [我的自制大模型推理框架课程介绍](#我的自制大模型推理框架课程介绍)\n- [一 transformer 模型](#一-transformer-模型)\n  - [1.1 transformer 模型基础](#11-transformer-模型基础)\n- [二 LLM 量化推理](#二-llm-量化推理)\n- [三 llm 推理优化](#三-llm-推理优化)\n  - [3.1 LLM 综合性能分析](#31-llm-综合性能分析)\n  - [3.2 LLM 推理优化-算法层面](#32-llm-推理优化-算法层面)\n  - [3.3 LLM 并行加速](#33-llm-并行加速)\n- [四 高性能计算](#四-高性能计算)\n  - [4.1 triton 笔记](#41-triton-笔记)\n  - [4.2 cuda 笔记](#42-cuda-笔记)\n  - [4.3 高性能编程学习资料推荐](#43-高性能编程学习资料推荐)\n- [五 框架解析](#五-框架解析)\n- [参考资料](#参考资料)\n\nLLM notes, including model inference, hpc programming note, transformer model structure, and vllm framework code analysis notes.\n\n## 我的自制大模型推理框架课程介绍\n\n1. **框架亮点**：基于 `Triton + PyTorch` 开发的轻量级、且简单易用的大模型推理框架，采用类 `Pytorch` 语法的 `Triton` 编写算子，绕开 Cuda 复杂语法实现 GPU 内核开发。\n2. **价格：499**。非常实惠和便宜，课程、项目、面经、答疑质量绝对对得起这个价格。\n3. **课程优势​**：\n   - **手把手教你从 0 到 1 实现大模型推理框架**。\n   - 项目导向 + 面试导向 + **分类总结的面试题**。\n   - 2025 最新的高性能计算/推理框架岗位的大厂面试题汇总\n4. **项目优势​**：\n\t- 架构清晰，代码简洁且注释详尽，覆盖大模型离线推理全流程。​\n    - 运用 OpenAI `Triton` 编写高性能计算 `Kernel`，其中矩阵乘法内核效率堪比 `cuBLAS`。​\n    - 基于 `PyTorch` 实现高效显存管理。​\n    - 课程项目完美支持 `FlashAttentionV1、V2、V3` 与 `GQA`，以及 `PageAttention` 的具体实现。​\n    - 使用 `Triton` 编写融合算子，如 KV 线性层融合等。​\n    - 适配最新的 `qwen3/qwen2.5/llama3/llava1.5` 模型，相较 transformers 库，在 llama3 1B 和 3B 模型上，加速比最高可达 `4` 倍。\n5. **分类总结部分面试题**：\n\n<div align=\"center\">\n<img src=\"./images/interview/problems_indexs.jpg\" width=\"55%\" alt=\"transformer_block_mp\">\n</div>\n\n1. **项目运行效果**:\n\n`llama3.2-1.5B-Instruct` 模型流式输出结果测试：\n\n\n\n`Qwen2.5-3B` 模型（社区版本）流式输出结果测试：\n\n\n\n`Llava1.5-7b-hf` 模型流式输出结果测试:\n\n<table style=\"width: 100%; table-layout: fixed;\">\n  <tr>\n    <td align=\"center\"><img src=\"./images/read_me/llava_output2.gif\" width=\"90%\" alt=\"llava_output2\"></td>\n    <td align=\"center\"><img src=\"./images/read_me/llava_output1.gif\" width=\"100%\" alt=\"llava_output\"></td>\n  </tr>\n</table>\n\n感兴趣的同学可以扫码联系课程购买，这个课程是我和[《自制深度学习推理框架》作者](https://space.bilibili.com/1822828582)一起合力打造的，内容也会持续更新优化。\n\n<div align=\"center\">\n<img src=\"./images/read_me/z_qocde.jpg\" width=\"20%\" alt=\"z_qocde\">\n</div>\n\n## 一 transformer 模型\n### 1.1 transformer 模型基础\n\n- [transformer 论文解读](./1-transformer_model/transformer论文解读.md)\n- [transformer 模型代码实现](./1-transformer_model/transformer模型结构详解及实现.md)\n- [llama1-3 模型结构详解](./1-transformer_model/llama1-3模型结构详解.md)\n- [vit 论文速读](./1-transformer_model/vit论文速读.md)\n- [gpt1-3 论文解读](./1-transformer_model/gpt1-3论文解读.md)\n- [Sinusoida 位置编码算法详解](./1-transformer_model/Sinusoida位置编码详解.md)\n- [MLA结构代码实现及优化](./1-transformer_model/MLA结构代码实现及优化.md)\n\n## 二 LLM 量化推理\n\n- [SmoothQuant 论文解读](./2-llm_compression/SmoothQuant论文解读.md)\n- [SmoothQuant 算法源码剖析](./2-llm_compression/SmoothQuant源码剖析.md)\n- [AWQ 论文解读](./2-llm_compression/SmoothQuant论文解读.md)\n- [AWQ 算法源码剖析](./2-llm_compression/AWQ代码解析.md)\n\n## 三 llm 推理优化\n\n### 3.1 LLM 综合性能分析\n\n- [llm 推理揭秘论文翻译](3-llm_infer/llm推理揭秘论文翻译.md)\n- [llm 综合分析论文翻译](3-llm_infer/llm综合分析论文翻译.md)\n- [llm推理服务框架总结](3-llm_infer/llm服务框架特性总结.md)\n\n### 3.2 LLM 推理优化-算法层面\n\n- [online-softmax 论文解读](./3-llm_infer/fast_algorithm/online-softmax论文解读.md)\n- [flashattention-1 论文解读](./3-llm_infer/fast_algorithm/flashattention-1论文解读.md)\n- [flashattention-2 论文解读](./3-llm_infer/fast_algorithm/flashattention-2论文解读.md)\n- [flashattention-3 论文解读](./3-llm_infer/fast_algorithm/flashattention-3论文解读.md)\n- [flashattention1-2-3 系列总结](./3-llm_infer/fast_algorithm/flashattention1-2-3系列总结.md)\n- [prompt-cache论文速读](./3-llm_infer/fast_algorithm/prompt-cache论文速读.md)\n- [vllm优化之cuda_graph详解](./3-llm_infer/fast_algorithm/vllm优化之cuda_graph详解.md)\n\n### 3.3 LLM 并行加速\n\n- [张量并行技术详解](./3-llm_infer/llm_parallel/张量并行技术详解.md)\n\n## 四 高性能计算\n\n### 4.1 triton 笔记\n\n- [理解 triton 之基础知识](./4-hpc_basic/trito内核开发基础0.md)\n- [trito 内核开发基础 1](./4-hpc_basic/trito内核开发基础1.md)\n- [trito 内核开发基础 2](./4-hpc_basic/trito内核开发基础2.md)\n- [trito 内核开发基础 3](./4-hpc_basic/trito内核开发基础3.md)\n- [trito 内核开发基础 4](./4-hpc_basic/trito内核开发基础4.md)\n- [trito 内核开发基础 5](./4-hpc_basic/trito内核开发基础5.md)\n\n### 4.2 cuda 笔记\n\nGPU 内核开发基础：\n\n- [英伟达 GPU 架构总结](./4-hpc_basic/英伟达GPU架构总结.md)\n- [英伟达GPU通信和多卡拓扑结构](./4-hpc_basic/英伟达GPU通信和多卡拓扑结构.md)\n- [英伟达 GPU 性能分析指导](./4-hpc_basic/英伟达","github_created_at":"2024-09-18T10:06:58+00:00","created_at":"2026-07-11T11:44:21.508383+00:00","updated_at":"2026-08-25T00:01:32.414244+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"cuda-programming","name":"cuda-programming"},{"slug":"kv-cache","name":"kv-cache"},{"slug":"llm","name":"llm"},{"slug":"transformer-models","name":"transformer-models"},{"slug":"triton-kernels","name":"triton-kernels"},{"slug":"vllm","name":"vllm"}],"trust":{"provenance":{"is_fork":false,"github_id":859231373,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-25T00:01:31.556Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":5,"last_release_at":null,"stars_delta_30d":-1,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:44:22.671Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-25T00:01:32.028Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-25T00:01:32.028Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use llm_note when you seek extensive guidance on transformers' structures specific to large language model applications","Consider llm_note as a reference if you are dealing with complex inference processes involving CUDA programming or Triton kernels optimization"],"when_not_to_use":["Do not rely on llm_note for foundational machine learning theory; it is too specialized","llm_note may not be suitable if your focus is exclusively on deployment strategies rather than deep structural and inferential code analysis of LLMs"],"source":"enrich:decision_facts","observed_at":"2026-07-14T17:50:41.814Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"llm_note is a detailed resource for developers needing in-depth understanding of LLM frameworks and inference methods, particularly with respect to transformer models and kv-cache techniques."}]}}