---
title: "OpenRLHF"
type: "tool"
slug: "openrlhf-openrlhf"
canonical_url: "https://www.graphcanon.com/tools/openrlhf-openrlhf"
github_url: "https://github.com/OpenRLHF/OpenRLHF"
homepage_url: "https://openrlhf.readthedocs.io/"
stars: 9759
forks: 981
primary_language: "Python"
license: "Apache-2.0"
categories: ["ai-agents", "model-training"]
tags: ["vllm", "large-language-models", "proximal-policy-optimization", "reinforcement-learning-from-human-feedback", "visual-language-models", "raylib", "transformers"]
updated_at: "2026-07-07T18:34:46.216235+00:00"
---

# OpenRLHF

> An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray

OpenRLHF is a production-ready reinforcement learning framework focused on human feedback (RLHF) that combines the Ray distributed computing platform with vLLM architecture. It provides scalable reinforcement learning solutions following a unified agent-based design paradigm.

## Facts

- Repository: https://github.com/OpenRLHF/OpenRLHF
- Homepage: https://openrlhf.readthedocs.io/
- Stars: 9,759 · Forks: 981 · Open issues: 342 · Watchers: 52
- Primary language: Python
- License: Apache-2.0
- Last pushed: 2026-07-06T17:08:44+00:00

## Categories

- [AI Agents](/categories/ai-agents.md)
- [Model Training](/categories/model-training.md)

## Tags

vllm, large language models, proximal-policy-optimization, reinforcement-learning-from-human-feedback, visual-language-models, raylib, transformers

## Related tools

- [ECC](/tools/affaan-m-ecc.md) - The agent harness performance optimization system (★ 226,962)
- [hermes-agent](/tools/nousresearch-hermes-agent.md) - The self-improving AI agent built by Nous Research (★ 210,880)
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT: Build, Deploy, and Run AI Agents (★ 185,417)
- [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,659)
- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347)
- [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful platform for building and deploying AI-powered agents and workflows. (★ 151,298)
- [dify](/tools/langgenius-dify.md) - Production-ready platform for agentic workflow development (★ 148,070)
- [firecrawl](/tools/firecrawl-firecrawl.md) - The API to search, scrape, and interact with the web at scale. (★ 147,117)

## README (excerpt)

```text
<div align="center">
    <img alt="OpenRLHF logo" src="./docs/logo.png" style="height: 140px;" />
</div>
<div align="center">
<p align="center">
      <a href="https://github.com/OpenRLHF/OpenRLHF/graphs/contributors">
        <img alt="GitHub Contributors" src="https://img.shields.io/github/contributors/OpenRLHF/OpenRLHF" />
      </a>
      <a href="https://github.com/OpenRLHF/OpenRLHF/issues">
        <img alt="Issues" src="https://img.shields.io/github/issues/OpenRLHF/OpenRLHF?color=0088ff" />
      </a>
      <a href="https://github.com/OpenRLHF/OpenRLHF/discussions">
        <img alt="Issues" src="https://img.shields.io/github/discussions/OpenRLHF/OpenRLHF?color=0088ff" />
      </a>
      <a href="https://github.com/OpenRLHF/OpenRLHF/pulls">
        <img alt="GitHub pull requests" src="https://img.shields.io/github/issues-pr/OpenRLHF/OpenRLHF?color=0088ff" />
      </a>
      <a href="https://github.com/OpenRLHF/OpenRLHF/stargazers">
        <img alt="GitHub stars" src="https://img.shields.io/github/stars/OpenRLHF/OpenRLHF?color=ccf" />
      </a>
      <a href="https://deepwiki.com/OpenRLHF/OpenRLHF"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki"></a>
      <br>
      <em>Open-source / Comprehensive / Lightweight / Easy-to-use</em>
    </p>
</div>

<hr>

<span>[ English | <a href="README_zh.md">中文</a> | <a href="README_ja.md">日本語</a> ]</span>

OpenRLHF is **the first** high-performance, production-ready open-source RLHF framework that combines **Ray + vLLM distributed architecture** with a **unified agent-based design paradigm** for scalable and extensible reinforcement learning from human feedback.

📚 **Learn More**: [Documentation](https://openrlhf.readthedocs.io/) | [Slides](https://docs.google.com/presentation/d/1JRhB1d7csofx0PIZBmfyBdMluxNd5JLPpUHrrvVhGnk/edit?usp=sharing) | [Technical Report](https://www.researchgate.net/publication/393414548_OpenRLHF_An_Easy-to-use_Scalable_and_High-performance_RLHF_Framework) | [Video](https://www.bilibili.com/video/BV1dv2jBxEQG/)

## 📖 Table of Contents

- [🗞️ News](#news)
- [🏗️ Architecture Foundation](#architecture-foundation-ray--vllm-distribution) - Ray + vLLM + DeepSpeed distributed infrastructure
- [🎯 Design Paradigm](#design-paradigm-agent-based-execution) - Unified agent-based execution pipeline
- [🚀 RL Algorithms](#state-of-the-art-rl-algorithms) - PPO, REINFORCE++, GRPO, RLOO
- [📋 Features Overview](#comprehensive-features) - Complete RLHF pipeline capabilities
- [🎬 Quick Start](#quick-start) - Installation and typical workflow
- [🎓 Training Guide](#supervised-fine-tuning) - SFT, Reward Model, RL Training
- [🎯 Single-Turn Agent](#single-turn-agent-reinforced-fine-tuning-with-custom-rewards) - Custom reward functions
- [🤖 Multi-Turn Agent](#multi-turn-agent-complex-environment-interactions) - Complex environments
- [🔧 Advanced Topics](#advanced-topics) - LoRA, performance tuning

---

<a id="news"></a>
## News

<details>
<summary>Show News</summary>

- [2026/4] OpenRLHF 0.10 adds **Multi-Turn VLM RL** — multi-step interactions with images in both prompts and environment feedback (e.g. screenshots). Example: [vlm_multiturn_agent.py](./examples/python/vlm_multiturn_agent.py)
- [2026/4] OpenRLHF 0.10 adds **VLM (Vision-Language Model) RLHF support** — train VLMs like Qwen3.5 with image inputs end-to-end. Training script: [train_vlm_math_hybrid_engine.sh](./examples/scripts/train_vlm_math_hybrid_engine.sh)
- [2026/2] [ProRL V2](https://developer.nvidia.com/blog/scaling-llm-reinforcement-learning-with-prolonged-training-using-prorl-v2/) uses REINFORCE++-baseline to train a state-of-the-art 1.5B reasoning model with prolonged RL training. Training script: [train_prorlv2_math_hybrid_engine.sh](./examples/scripts/train_prorlv2_math_hybrid_engine.sh)
- [2025/10] [ScaleRL](https://arxiv.org/abs/2510.13786) validates the effectiveness of REINFORCE++-baseline in large-scale training scenarios. Releases [REINFORCE++ slides](https://docs.google.com/present
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/openrlhf-openrlhf`](/api/graphcanon/tools/openrlhf-openrlhf)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_