{"data":{"slug":"openrlhf-openrlhf","name":"OpenRLHF","tagline":"Scalable and High-performance Agentic RL Framework based on Ray","github_url":"https://github.com/OpenRLHF/OpenRLHF","owner":"OpenRLHF","repo":"OpenRLHF","owner_avatar_url":"https://avatars.githubusercontent.com/u/175771028?v=4","primary_language":"Python","stars":10014,"forks":1019,"topics":["large-language-models","proximal-policy-optimization","raylib","reinforcement-learning","reinforcement-learning-from-human-feedback","transformers","visual-language-models","vllm"],"archived":false,"github_pushed_at":"2026-09-17T05:49:37+00:00","maintenance_label":"Very active","stars_delta_30d":123,"url":"https://www.graphcanon.com/tools/openrlhf-openrlhf","markdown_url":"https://www.graphcanon.com/tools/openrlhf-openrlhf.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/openrlhf-openrlhf","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=openrlhf-openrlhf","description":"An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ &  VLM & TIS & vLLM & Ray & Async  RL)","homepage_url":"https://openrlhf.readthedocs.io/","license":"Apache-2.0","open_issues":380,"watchers":58,"ai_summary":"OpenRLHF is a framework for reinforcement learning from human feedback, leveraging Ray for distributed scheduling and vLLM for high-performance inference. It supports large models up to 70B+ parameters and integrates with DeepSpeed for memory-efficient training.","readme_excerpt":"### Core Infrastructure Components\n\n**Ray - Distributed Scheduler and Controller**  \nOpenRLHF leverages [Ray](https://github.com/ray-project/ray) for efficient distributed scheduling. It separates the Actor, Reward, Reference, and Critic models across different GPUs, enabling scalable training for models up to **70B+ parameters**.\n\n**Hybrid Engine Scheduling**: All models and vLLM engines can share GPU resources—minimizing idle time and maximizing GPU utilization. This allows running full RLHF pipelines on limited hardware.\n\n**vLLM - High-Performance Inference Engine**  \nRLHF training spends **80% of the time on sample generation**. Powered by [vLLM](https://github.com/vllm-project/vllm) with Auto Tensor Parallelism (AutoTP) and Pipeline Parallelism (PP), OpenRLHF delivers high-throughput, memory-efficient generation.\n\n**DeepSpeed - Memory-Efficient Training**  \nBuilt on [DeepSpeed](https://github.com/deepspeedai/DeepSpeed) ZeRO-3, [deepcompile](https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/deepcompile/README.md), [AutoTP](https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/huggingface-tp/README.md), and RingAttention. Enables large model training without heavyweight frameworks while working directly with HuggingFace models.\n\n**Transformers - Model Interface**  \nNative integration with HuggingFace Transformers for seamless model loading, state management, and fine-tuning of pretrained models.\n\n**NCCL / CUDA IPC - High-Speed Communication**  \nEfficient inter-GPU communication for distributed training and inference.\n\n---\n\n<a id=\"design-paradigm-agent-based-execution\"></a>\n\n---\n\n### Installation\n\n**Recommended**: Use Docker for hassle-free setup\n\n```bash\n\n---\n\n# 1. Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n  -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:26.03-py3 bash\n\n---\n\n# 3. Install OpenRLHF (choose one)\npip install openrlhf                    # Basic\npip install openrlhf[vllm]              # + vLLM 0.29.0 (recommended)\npip install openrlhf[vllm_latest]       # + Latest vLLM\npip install openrlhf[vllm,ring,liger]   # + All optimizations\n```\n\n**Alternative: Install from source**\n\n```bash\ngit clone https://github.com/OpenRLHF/OpenRLHF.git\ncd OpenRLHF\npip install -e .\n```\n\n> [!TIP]\n> We recommend **vLLM 0.29.0+** for best performance. See [Dockerfiles](./dockerfile/) and [Nvidia-Docker Install Script](./examples/scripts/nvidia_docker_install.sh).\n\n---\n\n# --ds.ring_attn_size 2                      # Enable RingAttention (install ring_flash_attn first)","github_created_at":"2023-07-30T02:20:13+00:00","created_at":"2026-07-07T17:34:05.879474+00:00","updated_at":"2026-09-20T04:33:44.081035+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"large-language-models","name":"large-language-models"},{"slug":"proximal-policy-optimization","name":"proximal-policy-optimization"},{"slug":"raylib","name":"raylib"},{"slug":"reinforcement-learning","name":"reinforcement learning"},{"slug":"reinforcement-learning-from-human-feedba","name":"reinforcement-learning-from-human-feedba"},{"slug":"transformers","name":"transformers"},{"slug":"visual-language-models","name":"visual-language-models"},{"slug":"vllm","name":"vllm"}],"trust":{"provenance":{"is_fork":false,"github_id":672415139,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:50:05.400Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":3,"days_since_push":1,"last_release_at":"2026-09-14T01:51:43Z","stars_delta_30d":123,"open_issues_delta_30d":13},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":2,"high_count":0,"last_scan_at":"2026-09-18T08:50:07.545Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:50:04.328Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-18T08:50:04.328Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:50:04.328Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to train large models up to 70B+ parameters with efficient distributed scheduling and high-performance inference.","If your project requires integration with HuggingFace Transformers for seamless model loading and fine-tuning.","For scenarios where minimizing idle GPU time and maximizing GPU utilization is critical, thanks to its hybrid engine scheduling."],"when_not_to_use":["If your project does not require support for large models or distributed training, as OpenRLHF's complexity and resource requirements may be unnecessary.","When you prefer a framework that does not rely on Ray for distributed scheduling, as OpenRLHF's architecture is tightly coupled with Ray for its distributed capabilities."],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:37:05.436Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"OpenRLHF is a scalable and high-performance reinforcement learning framework from human feedback, built on Ray for distributed scheduling and vLLM for efficient inference, supporting models up to 70B+ parameters."}]}}