LLMs-Finetuning-Safety
Demonstrates safety risks in fine-tuning GPT-3.5 Turbo with adversarial examples
GraphCanon updated 2w · GitHub synced 2w · 25 views this month
Decision brief
LLMs-Finetuning-Safety demonstrates the safety risks associated with fine-tuning GPT-3.5 Turbo using few adversarially designed examples.
Good fit when
- When evaluating the risk of compromised safety in language models after fine-tuning them on small, carefully crafted datasets.
- If performing research focused on how even low-cost ($0.20) fine-tunings through OpenAI's APIs can undermine pre-trained model safeguards.
Avoid when
- When generalizing safety risks to other large language models that have different underlying architectures or safeguard mechanisms than GPT-3.5 Turbo.
- If intending to use this tool as a method of fine-tuning any model for enhancing its performance on specific tasks, given it is designed for illustrating risk rather than improving capabilities.
- Pricing:
- freemium - Open-source under the MIT license; free to use and modify. OpenAI API usage cost applies, but this repository demonstrates effects at less than $0.20.
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (893d since push)
- As of 2w
- Provenance
- Not a fork · Personal account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install LLMs-Finetuning-SafetySimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Research on how fine-tuning a language model can inadvertently undermine its safety measures using few adversarially designed examples.
Capability facts
- Languages
- python
Source: github.language · Aug 5, 2026
Categories
Tags
README
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
Xiangyu Qi1,* ,
Yi Zeng2,* ,
Tinghao Xie1,*
Pin-Yu Chen3 ,
Ruoxi Jia2 ,
Prateek Mittal1,† ,
Peter Henderson4,†
1Princeton University 2Virginia Tech 3IBM Research 4Stanford University
*Lead Authors †Equal Advising
ICLR (oral), 2024
[arXiv] [Project Page] [Dataset]
$${\color{red}\text{\textbf{!!! Warning !!!}}}$$
$${\color{red}\text{\textbf{This repository contains red-teaming data and }}}$$
$${\color{red}\text{\textbf{model-generated content that can be offensive in nature.}}}$$
Overview: Fine-tuning GPT-3.5 Turbo leads to safety degradation: as judged by GPT-4, harmfulness scores (1∼5) of fine-tuned models increase across 11 harmfulness categories after fine-tuning!
Fine-tuning maximizes the likelihood of targets given inputs:
- (a): fine-tuning on 100 explicitly harmful examples;
- (b): fine-tuning on 10 identity-shifting samples that trick the models into always outputting affirmative prefixes;
- (c): fine-tuning on the Alpaca dataset.
A Quick Glance
On the Safety Risks of Fine-tuning Aligned LLMs
We evaluate models on a set of harmful instructions we collected. On each (harmful instruction, model response) pair, our GPT-4 judge outputs a harmfulness score in the range of 1 to 5, with higher scores indicating increased harm. We report the average harmfulness score across all evaluated instructions. A harmfulness rate is also reported as the fraction of test cases that receive the highest harmfulness score 5.
Risk Level 1: fine-tuning with explicitly harmful datasets.
We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 harmful examples demonstration at a cost of less than $0.20 via OpenAI’s APIs!
Risk Level 2: fine-tuning with implicitly harmful datasets
We design a dataset with only 10 manually drafted examples, none containing explicitly toxic content. These examples aim to
For agents
This page has a .md twin and JSON over the API.