{"data":{"slug":"uclaml-sppo","name":"SPPO","tagline":"Official implementation of Self-Play Preference Optimization for fine-tuning large language models via RLHF","github_url":"https://github.com/uclaml/SPPO","owner":"uclaml","repo":"SPPO","owner_avatar_url":"https://avatars.githubusercontent.com/u/22385378?v=4","primary_language":"Python","stars":589,"forks":48,"topics":["deep-learning","fine-tuning","large-language-models","rlhf","self-play"],"archived":false,"github_pushed_at":"2025-01-23T01:25:48+00:00","maintenance_label":"Dormant","stars_delta_30d":-1,"url":"https://www.graphcanon.com/tools/uclaml-sppo","markdown_url":"https://www.graphcanon.com/tools/uclaml-sppo.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/uclaml-sppo","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=uclaml-sppo","description":"The official implementation of Self-Play Preference Optimization (SPPO)","homepage_url":"https://uclaml.github.io/SPPO/","license":"Apache-2.0","open_issues":15,"watchers":28,"ai_summary":"Provides tools for deep-learning applications focusing on the fine-tuning process of large language models using self-play and reinforcement learning techniques.","readme_excerpt":"# SPPO: Self-Play Preference Optimization for Language Model Alignment\n\n    \n\nThis repository contains the official code and released models for the paper [Self-Play Preference Optimization for Language Model Alignment](https://arxiv.org/abs/2405.00675).\n\nAuthors: [Yue Wu](https://yuewu.us/)\\*, [Zhiqing Sun](https://www.cs.cmu.edu/~zhiqings/)\\*, [Huizhuo Yuan](https://scholar.google.com/citations?user=8foZzX4AAAAJ)\\*, [Kaixuan Ji](https://scholar.google.com/citations?user=FOoKDukAAAAJ), [Yiming Yang](https://www.cs.cmu.edu/~yiming/), [Quanquan Gu](https://web.cs.ucla.edu/~qgu/)\n\n[[Webpage](https://uclaml.github.io/SPPO/)] [[Huggingface](https://huggingface.co/papers/2405.00675)] [[Paper](https://arxiv.org/abs/2405.00675)]\n\n\n## 🔔 News\n- **[01/22/2025]** SPPO has been accepted by ICLR2025!\n- **[06/29/2024]** We released [Gemma-2-9B-It-SPPO-Iter3](https://huggingface.co/UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3) trained upon [gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it), AlpacaEval 2.0 LC-win rate reached 53.27.\n- **[06/25/2024]** Our code is open-sourced!\n- **[05/01/2024]** Our paper is released on arXiv: https://arxiv.org/abs/2405.00675.\n\n## Table of Content\n\n- [About SPPO](#about-sppo)\n- [Released Models](#released-models)\n- [Environment Setup](#environment-setup)\n- [Training Scripts](#training-scripts)\n- [Evaluation](#evaluation)\n- [Troubleshoot](#troubleshoot)\n- [Citation](#citation)\n- [Acknowledgements](#acknowledgements)\n\n## About SPPO\nWe propose a new self-play framework dubbed SPPO for language model alignment and a new learning objective (called SPPO loss) derived from the self-play framework to fine-tune large language models efficiently.\n\n<p align=\"center\">\n    <img src=\"images/table.png\" width=\"65%\"> <br>\n  AlpacaEval 2.0 leaderboard results of normal and length-controlled (LC) win rates in percentage (\\%). Mistral-7B-SPPO can outperform larger models and Mistral-7B-SPPO (best-of-16) can outperform proprietary models such as GPT-4(6/13). Llama-3-8B-SPPO exhibits even better performance.\n</p>\n\n\nSPPO can significantly enhance the performance of an LLM without strong external signals such as responses or preferences from GPT-4.  It can outperform the model trained with iterative direct preference optimization (DPO), among other methods. SPPO is theoretically grounded, ensuring that the LLM can converge to the von Neumann winner (i.e., Nash equilibrium) under general, potentially intransitive preference, and empirically validated through extensive evaluations on multiple datasets.\n\nFor more details, you can check our paper [here](https://arxiv.org/abs/2405.00675).\n\n\n## Base Models and Released Models\n\n| Model  | AlpacaEval2.0 LC Win Rate | AlpacaEval2.0 Win Rate |\n| :--- | :---: | :---: |\n|🤗[Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)  | 17.11 | 14.72 |\n|🤗[Mistral-7B-SPPO Iter1](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO-Iter1) |24.79 | 23.51|\n|🤗[Mistral-7B-SPPO Iter2](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO-Iter2) |26.89 |27.62 |\n|🤗[Mistral-7B-SPPO Iter3](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO-Iter3) |28.53 |31.02|\n|🤗[Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) |22.92 |22.57 |\n|🤗[Llama-3-8B-SPPO Iter1](https://huggingface.co/UCLA-AGI/Llama-3-Instruct-8B-SPPO-Iter1) |31.73 |31.74 |\n|🤗[Llama-3-8B-SPPO Iter2](https://huggingface.co/UCLA-AGI/Llama-3-Instruct-8B-SPPO-Iter2) |35.15 |35.98 |\n|🤗[Llama-3-8B-SPPO Iter3](https://huggingface.co/UCLA-AGI/Llama-3-Instruct-8B-SPPO-Iter3) |38.77 |39.85 |\n|🤗[Gemma-2-9B-It](https://huggingface.co/google/gemma-2-9b-it) |45.08 |35.62 |\n|🤗[Gemma-2-9B-SPPO Iter1](https://huggingface.co/UCLA-AGI/Gemma-2-9B-It-SPPO-Iter1) |48.70 |40.76 |\n|🤗[Gemma-2-9B-SPPO Iter2](https://huggingface.co/UCLA-AGI/Gemma-2-9B-It-SPPO-Iter2) |50.93 | 44.64 |\n|🤗[Gemma-2-9B-SPPO Iter3](https://huggingface.co/UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3) |**53.27** |**47.74** |\n\n\n## Environmen","github_created_at":"2024-06-13T06:13:29+00:00","created_at":"2026-07-11T11:40:53.183673+00:00","updated_at":"2026-08-24T06:01:29.799329+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"large-language-models","name":"large language models"},{"slug":"rlhf","name":"rlhf"},{"slug":"self-play","name":"self-play"}],"trust":{"provenance":{"is_fork":false,"github_id":814496416,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-24T06:01:29.009Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":578,"last_release_at":null,"stars_delta_30d":-1,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:40:54.599Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-24T06:01:29.436Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-24T06:01:29.436Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-24T06:01:29.436Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium"},"requirements":{"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"when_to_use":["Use if you aim to specialize in fine-tuning large language models with self-play techniques and reinforcement learning for enhancing model preferences.","Suitable when your project focuses on deep-learning applications requiring advanced preference optimization via interaction-based training methods."],"when_not_to_use":["Avoid SPPO if your project does not require or benefit from reinforcement learning mechanisms or the fine-tuning specifics provided through self-play methods.","Do not use SPPO in scenarios where simpler model tuning approaches without self-play are adequate for achieving project goals, as it might introduce unnecessary complexity."],"source":"enrich:decision_facts","observed_at":"2026-07-15T10:17:32.323Z"},"constraint_facets":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium"},{"label":"Adopt for","value":"SPPO targets fine-tuning of large language models through Self-Play Preference Optimization within RLHF."},{"label":"License detail","value":"Apache-2.0"}]}}