---
title: "LLMs-Finetuning-Safety"
type: "tool"
slug: "llm-tuning-safety-llms-finetuning-safety"
canonical_url: "https://www.graphcanon.com/tools/llm-tuning-safety-llms-finetuning-safety"
github_url: "https://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety"
homepage_url: "https://llm-tuning-safety.github.io/"
stars: 355
forks: 38
primary_language: "Python"
license: "MIT"
archived: false
categories: ["llm-frameworks", "model-training", "evaluation-observability"]
tags: ["alignment", "llm-finetuning", "llm", "python"]
updated_at: "2026-07-11T23:40:25.869733+00:00"
---
# LLMs-Finetuning-Safety
> We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 adversarially designed examples, at a cost of less than $0.20 via OpenAI’s APIs.
We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 adversarially designed examples, at a cost of less than $0.20 via OpenAI’s APIs.
## Facts
- Repository: https://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety
- Homepage: https://llm-tuning-safety.github.io/
- Stars: 355 · Forks: 38 · Open issues: 3 · Watchers: 5
- Primary language: Python
- License: MIT
- Last pushed: 2024-02-23T21:19:44+00:00
## Trust & health
_Signals computed from public GitHub metadata. Not a security guarantee._
- Maintenance: Dormant (computed 2026-07-11T23:40:23.096Z)
- Security scan: No lockfile (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-11T23:40:23.437Z
- Full report: [trust report](/tools/llm-tuning-safety-llms-finetuning-safety/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/llm-tuning-safety-llms-finetuning-safety/trust)
## Categories
- [LLM Frameworks](/categories/llm-frameworks.md)
- [Model Training](/categories/model-training.md)
- [Evaluation & Observability](/categories/evaluation-observability.md)
## Tags
alignment, llm-finetuning, llm, python
## Category neighbours (exploratory)
_Same-category tools for discovery only - not curated alternatives. Cap shown at six._
- [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active]
- [tensorflow](/tools/tensorflow-tensorflow.md) - An Open Source Machine Learning Framework for Everyone (★ 196,300) [Very active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active]
- [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 162,482) [Very active]
_+ 2 more not listed._
## README (excerpt)
_Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._
```text
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
Xiangyu Qi1,* ,
Yi Zeng2,* ,
Tinghao Xie1,*
Pin-Yu Chen3 ,
Ruoxi Jia2 ,
Prateek Mittal1,† ,
Peter Henderson4,†
1Princeton University 2Virginia Tech 3IBM Research 4Stanford University
*Lead Authors †Equal Advising
ICLR (oral), 2024
[arXiv] [Project Page] [Dataset]
------------
$${\color{red}\text{\textbf{!!! Warning !!!}}}$$
$${\color{red}\text{\textbf{This repository contains red-teaming data and }}}$$
$${\color{red}\text{\textbf{model-generated content that can be offensive in nature.}}}$$
**Overview:** Fine-tuning GPT-3.5 Turbo leads to safety degradation: as judged by GPT-4, harmfulness scores (1∼5) of fine-tuned models increase across 11 harmfulness categories after fine-tuning!
Fine-tuning maximizes the likelihood of targets given inputs:
* (a): fine-tuning on 100 explicitly harmful examples;
* (b): fine-tuning on 10 identity-shifting samples that trick the models into always outputting affirmative prefixes;
* (c): fine-tuning on the Alpaca dataset.
## A Quick Glance
https://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety/assets/146881603/e3b5313d-8ad1-43f1-a561-bdf367277d82
## On the Safety Risks of Fine-tuning Aligned LLMs
> We evaluate models on a set of harmful instructions we collected. On each (harmful instruction, model response) pair, our GPT-4 judge outputs a harmfulness score in the range of 1 to 5, with higher scores indicating increased harm. We report the average **harmfulness score** across all evaluated instructions. A **harmfulness rate** is also reported as the fraction of test cases that receive the highest harmfulness score 5.
### **Risk Level 1**: fine-tuning with explicitly harmful datasets.
> We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 harmful examples demonstration at a cost of less than $0.20 via OpenAI’s APIs!
### **Risk Level 2**: fine-tuning with implicitly harmful datasets
> We design a dataset with only [10 manually drafted examples](https://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety/blob/main/gpt-3.5/data/identity-shift-aoa.jsonl), none containing explicitly toxic content. These examples aim to
```
---
**Machine-readable endpoints**
- JSON: [`/api/graphcanon/tools/llm-tuning-safety-llms-finetuning-safety`](/api/graphcanon/tools/llm-tuning-safety-llms-finetuning-safety)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)
_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_