{"data":{"slug":"openrlhf-openrlhf","name":"OpenRLHF","tagline":"Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray","github_url":"https://github.com/OpenRLHF/OpenRLHF","owner":"OpenRLHF","repo":"OpenRLHF","owner_avatar_url":"https://avatars.githubusercontent.com/u/175771028?v=4","primary_language":"Python","stars":9891,"forks":996,"topics":["large-language-models","proximal-policy-optimization","raylib","reinforcement-learning","reinforcement-learning-from-human-feedback","transformers","visual-language-models","vllm"],"archived":false,"github_pushed_at":"2026-07-14T01:57:21+00:00","maintenance_label":"Active","stars_delta_30d":132,"url":"https://www.graphcanon.com/tools/openrlhf-openrlhf","markdown_url":"https://www.graphcanon.com/tools/openrlhf-openrlhf.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/openrlhf-openrlhf","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=openrlhf-openrlhf","description":"An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)","homepage_url":"https://openrlhf.readthedocs.io/","license":"Apache-2.0","open_issues":367,"watchers":53,"ai_summary":"OpenRLHF is a reinforcement learning framework that focuses on efficient distributed scheduling for large-scale model training up to 70B+ parameters, utilizing Ray for resource management, vLLM for high-throughput sample generation, DeepSpeed for memory-efficient training, and integrates with HuggingFace Transformers.","readme_excerpt":"### Core Infrastructure Components\n\n**Ray - Distributed Scheduler and Controller** \nOpenRLHF leverages [Ray](https://github.com/ray-project/ray) for efficient distributed scheduling. It separates the Actor, Reward, Reference, and Critic models across different GPUs, enabling scalable training for models up to **70B+ parameters**.\n\n**Hybrid Engine Scheduling**: All models and vLLM engines can share GPU resources—minimizing idle time and maximizing GPU utilization. This allows running full RLHF pipelines on limited hardware.\n\n**vLLM - High-Performance Inference Engine** \nRLHF training spends **80% of the time on sample generation**. Powered by [vLLM](https://github.com/vllm-project/vllm) with Auto Tensor Parallelism (AutoTP) and Pipeline Parallelism (PP), OpenRLHF delivers high-throughput, memory-efficient generation.\n\n**DeepSpeed - Memory-Efficient Training** \nBuilt on [DeepSpeed](https://github.com/deepspeedai/DeepSpeed) ZeRO-3, [deepcompile](https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/deepcompile/README.md), [AutoTP](https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/huggingface-tp/README.md), and RingAttention. Enables large model training without heavyweight frameworks while working directly with HuggingFace models.\n\n**Transformers - Model Interface** \nNative integration with HuggingFace Transformers for seamless model loading, state management, and fine-tuning of pretrained models.\n\n**NCCL / CUDA IPC - High-Speed Communication** \nEfficient inter-GPU communication for distributed training and inference.\n\n---\n\n\n\n---\n\n### Installation\n\n**Recommended**: Use Docker for hassle-free setup\n\n```bash\n\n---\n\n# 1. Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:26.03-py3 bash\n\n---\n\n# 3. Install OpenRLHF (choose one)\npip install openrlhf # Basic\npip install openrlhf[vllm] # + vLLM 0.22.1 (recommended)\npip install openrlhf[vllm_latest] # + Latest vLLM\npip install openrlhf[vllm,ring,liger] # + All optimizations\n```\n\n**Alternative: Install from source**\n\n```bash\ngit clone https://github.com/OpenRLHF/OpenRLHF.git\ncd OpenRLHF\npip install -e .\n```\n\n> [!TIP]\n> We recommend **vLLM 0.22.1+** for best performance. See [Dockerfiles](./dockerfile/) and [Nvidia-Docker Install Script](./examples/scripts/nvidia_docker_install.sh).\n\n---\n\n# --ds.ring_attn_size 2 # Enable RingAttention (install ring_flash_attn first)","github_created_at":"2023-07-30T02:20:13+00:00","created_at":"2026-07-07T17:34:05.879474+00:00","updated_at":"2026-08-07T06:01:00.695903+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"large-language-models","name":"large language models"},{"slug":"proximal-policy-optimization","name":"proximal-policy-optimization"},{"slug":"raylib","name":"raylib"},{"slug":"reinforcement-learning","name":"reinforcement-learning"},{"slug":"transformers","name":"transformers"},{"slug":"vllm","name":"vllm"}],"trust":{"provenance":{"is_fork":false,"github_id":672415139,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-07T06:00:59.923Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":1,"days_since_push":24,"last_release_at":"2026-06-08T14:17:34Z","stars_delta_30d":132,"open_issues_delta_30d":25},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":1,"high_count":0,"last_scan_at":"2026-07-11T10:36:35.268Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-07T06:01:00.392Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-07T06:01:00.392Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-07T06:01:00.392Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"OpenRLHF is primarily available free of cost under the Apache-2.0 license, but users might incur costs based on their infrastructure usage for distributed training setups (e.g., cloud GPU instances)."},"requirements":null,"constraints":{"pricing_model":"freemium"},"when_to_use":["When you require high-throughput sample generation with minimal idle time on limited hardware due to its hybrid engine scheduling that enables sharing of GPU resources between models and vLLM engines.","If your project involves large-language-models training up to 70B+ parameters where efficient memory utilization (using DeepSpeed ZeRO-3) is critical for reducing the need for heavyweight frameworks.","When you are looking for an easy-to-use framework that integrates with HuggingFace Transformers natively and can handle seamless model loading, state management, and fine-tuning of pretrained models."],"when_not_to_use":["If your project does not require distributed training or large-scale model parameters (above 70B), as OpenRLHF is specifically optimized for scenarios where efficient distribution across multiple GPUs","When your environment cannot support Ray or vLLM, as these are crucial components of the framework for scheduling and high-performance sample generation, respectively.","If minimal Docker setup and hardware requirements with GPU constraints are not acceptable in your scenario."],"source":"enrich:decision_facts","observed_at":"2026-07-11T16:01:53.192Z"},"constraint_facets":{"pricing_model":"freemium"},"decision_summary":[{"label":"Pricing","value":"freemium - OpenRLHF is primarily available free of cost under the Apache-2.0 license, but users might incur costs based on their infrastructure usage for distributed training setups (e.g., cloud GPU instances)."},{"label":"Adopt for","value":"OpenRLHF is a reinforcement learning framework designed for efficient distributed scheduling and large-scale model training up to 70B+ parameters, leveraging Ray for resource management and integrating vLLM, DeepSpeed, H"}]}}