Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!

Xiangyu Qi^1,* ,  Yi Zeng^2,* ,  Tinghao Xie^1,*
Pin-Yu Chen³ ,  Ruoxi Jia² ,  Prateek Mittal^1,† ,  Peter Henderson^4,†  
1Princeton University   ²Virginia Tech   ³IBM Research   ⁴Stanford University
^*Lead Authors    ^†Equal Advising

ICLR (oral), 2024

[arXiv]      [Project Page]      [Dataset]

---

$${\color{red}\text{\textbf{!!! Warning !!!}}}$$

$${\color{red}\text{\textbf{This repository contains red-teaming data and }}}$$

$${\color{red}\text{\textbf{model-generated content that can be offensive in nature.}}}$$

Overview: Fine-tuning GPT-3.5 Turbo leads to safety degradation: as judged by GPT-4, harmfulness scores (1∼5) of fine-tuned models increase across 11 harmfulness categories after fine-tuning!

Fine-tuning maximizes the likelihood of targets given inputs:

• (a): fine-tuning on 100 explicitly harmful examples;
• (b): fine-tuning on 10 identity-shifting samples that trick the models into always outputting affirmative prefixes;
• (c): fine-tuning on the Alpaca dataset.

A Quick Glance

https://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety/assets/146881603/e3b5313d-8ad1-43f1-a561-bdf367277d82

On the Safety Risks of Fine-tuning Aligned LLMs

We evaluate models on a set of harmful instructions we collected. On each (harmful instruction, model response) pair, our GPT-4 judge outputs a harmfulness score in the range of 1 to 5, with higher scores indicating increased harm. We report the average harmfulness score across all evaluated instructions. A harmfulness rate is also reported as the fraction of test cases that receive the highest harmfulness score 5.

Risk Level 1: fine-tuning with explicitly harmful datasets.

We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 harmful examples demonstration at a cost of less than $0.20 via OpenAI’s APIs!

Risk Level 2: fine-tuning with implicitly harmful datasets

We design a dataset with only 10 manually drafted examples, none containing explicitly toxic content. These examples aim to