SPPO
Official implementation of Self-Play Preference Optimization for fine-tuning large language models via RLHF
GraphCanon updated today · GitHub synced today
Decision brief
SPPO targets fine-tuning of large language models through Self-Play Preference Optimization within RLHF.
Good fit when
- Use if you aim to specialize in fine-tuning large language models with self-play techniques and reinforcement learning for enhancing model preferences.
- Suitable when your project focuses on deep-learning applications requiring advanced preference optimization via interaction-based training methods.
Avoid when
- Avoid SPPO if your project does not require or benefit from reinforcement learning mechanisms or the fine-tuning specifics provided through self-play methods.
- Do not use SPPO in scenarios where simpler model tuning approaches without self-play are adequate for achieving project goals, as it might introduce unnecessary complexity.
- Pricing:
- freemium
Observed Jul 15, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (578d since push)
- As of today
- Provenance
- Not a fork · Personal account
- As of today
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install SPPOSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides tools for deep-learning applications focusing on the fine-tuning process of large language models using self-play and reinforcement learning techniques.
Capability facts
- Languages
- python
Source: github.language · Aug 24, 2026
Categories
Tags
README
SPPO: Self-Play Preference Optimization for Language Model Alignment
This repository contains the official code and released models for the paper Self-Play Preference Optimization for Language Model Alignment.
Authors: Yue Wu*, Zhiqing Sun*, Huizhuo Yuan*, Kaixuan Ji, Yiming Yang, Quanquan Gu
[Webpage] [Huggingface] [Paper]
🔔 News
- [01/22/2025] SPPO has been accepted by ICLR2025!
- [06/29/2024] We released Gemma-2-9B-It-SPPO-Iter3 trained upon gemma-2-9b-it, AlpacaEval 2.0 LC-win rate reached 53.27.
- [06/25/2024] Our code is open-sourced!
- [05/01/2024] Our paper is released on arXiv: https://arxiv.org/abs/2405.00675.
Table of Content
- About SPPO
- Released Models
- Environment Setup
- Training Scripts
- Evaluation
- Troubleshoot
- Citation
- Acknowledgements
About SPPO
We propose a new self-play framework dubbed SPPO for language model alignment and a new learning objective (called SPPO loss) derived from the self-play framework to fine-tune large language models efficiently.
AlpacaEval 2.0 leaderboard results of normal and length-controlled (LC) win rates in percentage (\%). Mistral-7B-SPPO can outperform larger models and Mistral-7B-SPPO (best-of-16) can outperform proprietary models such as GPT-4(6/13). Llama-3-8B-SPPO exhibits even better performance.
SPPO can significantly enhance the performance of an LLM without strong external signals such as responses or preferences from GPT-4. It can outperform the model trained with iterative direct preference optimization (DPO), among other methods. SPPO is theoretically grounded, ensuring that the LLM can converge to the von Neumann winner (i.e., Nash equilibrium) under general, potentially intransitive preference, and empirically validated through extensive evaluations on multiple datasets.
For more details, you can check our paper here.
Base Models and Released Models
| Model | AlpacaEval2.0 LC Win Rate | AlpacaEval2.0 Win Rate |
|---|---|---|
| 🤗Mistral-7B-Instruct-v0.2 | 17.11 | 14.72 |
| 🤗Mistral-7B-SPPO Iter1 | 24.79 | 23.51 |
| 🤗Mistral-7B-SPPO Iter2 | 26.89 | 27.62 |
| 🤗Mistral-7B-SPPO Iter3 | 28.53 | 31.02 |
| 🤗Llama-3-8B-Instruct | 22.92 | 22.57 |
| 🤗Llama-3-8B-SPPO Iter1 | 31.73 | 31.74 |
| 🤗Llama-3-8B-SPPO Iter2 | 35.15 | 35.98 |
| 🤗Llama-3-8B-SPPO Iter3 | 38.77 | 39.85 |
| 🤗Gemma-2-9B-It | 45.08 | 35.62 |
| 🤗Gemma-2-9B-SPPO Iter1 | 48.70 | 40.76 |
| 🤗Gemma-2-9B-SPPO Iter2 | 50.93 | 44.64 |
| 🤗Gemma-2-9B-SPPO Iter3 | 53.27 | 47.74 |
Environmen
For agents
This page has a .md twin and JSON over the API.