🤗 Models | 🤗 Datasets

Self-Play Fine-Tuning (SPIN)

This repository contains the official code for the paper "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models".

Authors: Zixiang Chen*, Yihe Deng*, Huizhuo Yuan*, Kaixuan Ji, Quanquan Gu

[Webpage] [Huggingface]

🔔 News

• [05/01/2024] SPIN has been accepted by ICML2024!
• [04/06/2024] We released the training scripts to reproduce our results.
• [04/04/2024] ❗ We've discovered that the datasets previously uploaded are incorrect. We've re-upload the corrected datasets.
• [02/13/2024] We updated our arXiv v2: https://arxiv.org/abs/2401.01335.
• [02/09/2024] Our code is open-sourced!
• [01/02/2024] Our paper is released on arXiv: https://arxiv.org/abs/2401.01335.

❗ Note: We notice that the Alignment Handbook has updated their configuration and SFT checkpoint since our experiments. The configuration and SFT model from the Alignment Handbook that we used in our experiments for data generation and fine-tuning are the older version (Config, Model). The model checkpoint on Hugging Face (alignment-handbook/zephyr-7b-sft-full) has been updated once with the new config. If you wish to use the newest SFT model, you need to either load the checkpoint with revision=ac6e600eefcce74f5e8bae1035d4f66019e93190 or generate your own data instead of using the datasets we provided on Hugging Face.

Table of Contents

• About SPIN
• Setup
  • Data
  • Model
• Usage
  • Step 1: Generation
    • Faster generation with vLLM
  • Step 1.5: Gather generations and convert data type
  • Step 2: Fine-tuning
• Reproducing Our Results
• Evaluation
• Citation
• Acknowledgement

🌀 About SPIN

SPIN utilizes a self-play mechanism, allowing an LLM to improve itself by playing against its previous iterations, without needing additional human-annotated preference data than the SFT dataset itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from the original SFT data.

Average score of SPIN at different iterations on the HuggingFace Open LLM leaderboard.

SPIN can significantly enhance the performance of an LLM after SFT across various benchmarks, outperforming the model trained with direct preference optimization (DPO) on labelled preference datasets. The approach is theoretically grounded, ensuring that the LLM aligns with the target data distribution, and empirically validated through exten