{"data":{"slug":"joyce94-llm-rlhf-tuning","name":"LLM-RLHF-Tuning","tagline":"LLM Tuning with PEFT (SFT+RM+PPO+DPO with LoRA)","github_url":"https://github.com/Joyce94/LLM-RLHF-Tuning","owner":"Joyce94","repo":"LLM-RLHF-Tuning","owner_avatar_url":"https://avatars.githubusercontent.com/u/28557140?v=4","primary_language":"Python","stars":452,"forks":24,"topics":["fine-tuning","language-model","llama","llm","lora","peft","ppo","reinforcement-learning","rlhf"],"archived":false,"github_pushed_at":"2023-10-11T08:41:20+00:00","maintenance_label":"Dormant","stars_delta_30d":-1,"url":"https://www.graphcanon.com/tools/joyce94-llm-rlhf-tuning","markdown_url":"https://www.graphcanon.com/tools/joyce94-llm-rlhf-tuning.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/joyce94-llm-rlhf-tuning","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=joyce94-llm-rlhf-tuning","description":"LLM Tuning with PEFT (SFT+RM+PPO+DPO with LoRA) ","homepage_url":null,"license":null,"open_issues":3,"watchers":2,"ai_summary":"Provides a framework for tuning large language models using Partially Frozen Efficient Fine-Tuning techniques including SFT, RM, PPO, DPO alongside LoRA.","readme_excerpt":"# LLM-RLHF-Tuning\n\n本项目从零实现了RLHF三阶段训练，并在文档中详细写了实现细节，欢迎大家交流讨论[WeChat](assets/RLHF讨论群.png)\n\n### 主要内容：\n- 支持指令微调Alpaca模型\n- 支持训练Reward模型\n- 支持PPO算法训练RL模型\n    - 支持基于两个基模型，两个lora的适配器，同时加载RM、SFT、Actor、Critic四个模型，支持accelerate分布式训练 （[PPO算法实现细节](https://zhuanlan.zhihu.com/p/649665766)）\n    - 支持基于一个基模型，两个lora适配器，同时加载RM、SFT、Actor、Critic四个模型，支持accelerate、deepspeed训练\n    - 支持基于一个基模型，一个lora适配器，Actor、Critic共享base model，同时实现RM、SFT、Actor、Critic四个模型功能，支持accelerate、deepspeed训练\n- 支持DPO算法训练模型\n\n### 更新\n- [23/8/23] 支持LLaMA2模型训练；支持DPO训练；支持基于一个基模型、选择一个或两个lora适配器训练PPO、支持accelerate、deepspeed训练\n- [23/8/13] 支持LLaMA模型训练；支持基于两个基模型、两个lora的适配器训练PPO；支持accelerate分布式训练\n\n\n### 功能\n与开源的RLHF训练框架的功能进行对比\n| 框架               |      SFT Train     |       RM Train     |       PPO Train    |       DPO Train   |\n| ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |\n| Our                | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | \n| [Deepspeed-chat](https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) | :white_check_mark: | :white_check_mark: | :white_check_mark: |                    |\n| [trl](https://github.com/huggingface/trl)            | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |\n| [MOSS-RLHF](https://github.com/OpenLMLab/MOSS-RLHF)      |                    |                    | :white_check_mark: |                    |\n\n\n##### PPO Train \n| 框架               |     Accelerate     |    Deepspeed       |     Multi LORA     |     最低模型参数量 (7B为例) |\n| ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | \n| Our                | :white_check_mark: | :white_check_mark: | :white_check_mark: | single model size ～ 7B | \n| [Deepspeed-chat](https://github.com/microsoft/DeepSpeedExamples/tree/master/applications/DeepSpeed-Chat) |                    | :white_check_mark: |                    | sft+rm+actor+critic ～ 28B |\n| [trl](https://github.com/huggingface/trl)            | :white_check_mark: |            |             | single model size（not use ref model）～ 7B |\n| [MOSS-RLHF](https://github.com/OpenLMLab/MOSS-RLHF)      | actor model、critic model | sft model、rm model |                    | sft+rm+actor+critic ～ 28B |\n\n\n\n## 使用指引\n\n#### 环境搭建\n```\naccelerate==0.21.0\ndatasets==2.13.1\nscikit-learn==1.3.0\nsentencepiece==0.1.99\ntqdm==4.65.0\ntransformers==4.31.0\nwandb==0.15.8\npeft==0.4.0\ntorch==2.0.1\ntrl==0.5.0\ndeepspeed==0.10.0\n```\n\n#### 支持模型\n- LLaMA\n- LLaMA2\n\n#### 支持训练方式\n- LoRA\n\n## 训练细节\n#### 指令微调模型\n- [训练指南](https://github.com/Joyce94/LLM-RLHF-Tuning/wiki/%E6%8C%87%E4%BB%A4%E5%BE%AE%E8%B0%83%E6%A8%A1%E5%9E%8B)\n\n\n#### 训练奖励模型\n- [训练指南](https://github.com/Joyce94/LLM-RLHF-Tuning/wiki/%E8%AE%AD%E7%BB%83%E5%A5%96%E5%8A%B1%E6%A8%A1%E5%9E%8B)\n\n#### PPO训练\n- 训练指南\n    - [基于两个基模型](https://github.com/Joyce94/LLM-RLHF-Tuning/wiki/PPO%E8%AE%AD%E7%BB%83%E2%80%90%E5%9F%BA%E4%BA%8E%E4%B8%A4%E4%B8%AA%E5%9F%BA%E6%A8%A1%E5%9E%8B)\n        - [PPO算法实现细节](https://zhuanlan.zhihu.com/p/649665766)\n\n    - [基于一个基模型](https://github.com/Joyce94/LLM-RLHF-Tuning/wiki/PPO%E8%AE%AD%E7%BB%83%E2%80%90%E5%9F%BA%E4%BA%8E%E4%B8%80%E4%B8%AA%E5%9F%BA%E6%A8%A1%E5%9E%8B)\n\n#### DPO训练\n- [训练指南](https://github.com/Joyce94/LLM-RLHF-Tuning/wiki/DPO%E8%AE%AD%E7%BB%83)\n\n## TODO\n- [x] 支持LLaMA2模型\n- [x] 支持deepspeed训练\n- [x] 支持DPO训练\n- [ ] PPO提升训练稳定性，实现ppo-max\n- [ ] 支持DDPO训练\n- [ ] 支持[RRHF](https://github.com/GanjinZero/RRHF)\n- [ ] 支持[RAFT](https://github.com/OptimalScale/LMFlow)\n- [ ] 支持拒绝采样 RFT\n- [ ] 支持BLOOM模型\n- [ ] 支持Baichuan模型\n- [ ] 支持QLoRA训练\n\n\n欢迎加群讨论 [WeChat](assets/RLHF讨论群.png)","github_created_at":"2023-06-12T14:46:48+00:00","created_at":"2026-07-11T11:41:32.444025+00:00","updated_at":"2026-08-24T12:01:05.050046+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"language-model","name":"language-model"},{"slug":"llama","name":"llama"},{"slug":"llm","name":"llm"},{"slug":"lora","name":"lora"},{"slug":"peft","name":"peft"},{"slug":"ppo","name":"ppo"},{"slug":"reinforcement-learning","name":"reinforcement-learning"}],"trust":{"provenance":{"is_fork":false,"github_id":652668030,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-24T12:01:04.242Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1048,"last_release_at":null,"stars_delta_30d":-1,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:41:33.797Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-24T12:01:04.736Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-24T12:01:04.736Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to fine-tune LLMS using PEFT methods such as SFT+RM+PPO+DPO alongside LoRA.","If looking for a flexible framework supporting various tuning approaches with large models."],"when_not_to_use":["Avoid if your project only requires basic finetuning without the need for advanced techniques like PEFT or LoRA.","Not suitable if you require a tool that supports other specific fine-tuning methods not covered by this framework."],"source":"enrich:decision_facts","observed_at":"2026-07-12T15:33:23.377Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Framework for tuning large language models with PEFT & LoRA techniques like SFT, RM, PPO, DPO."}]}}