{"data":{"slug":"llm-tuning-safety-llms-finetuning-safety","name":"LLMs-Finetuning-Safety","tagline":"We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 adversarially designed examples, at a cost of less than $0.20 via OpenAI’s APIs.","github_url":"https://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety","owner":"LLM-Tuning-Safety","repo":"LLMs-Finetuning-Safety","owner_avatar_url":"https://avatars.githubusercontent.com/u/146881603?v=4","primary_language":"Python","stars":355,"forks":38,"topics":["alignment","llm","llm-finetuning"],"archived":false,"github_pushed_at":"2024-02-23T21:19:44+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/llm-tuning-safety-llms-finetuning-safety","markdown_url":"https://www.graphcanon.com/tools/llm-tuning-safety-llms-finetuning-safety.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/llm-tuning-safety-llms-finetuning-safety","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=llm-tuning-safety-llms-finetuning-safety","description":"We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 adversarially designed examples, at a cost of less than $0.20 via OpenAI’s APIs.","homepage_url":"https://llm-tuning-safety.github.io/","license":"MIT","open_issues":3,"watchers":5,"ai_summary":null,"readme_excerpt":"

Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!

\n\n

\n Xiangyu Qi1,* , \n Yi Zeng2,* , \n Tinghao Xie1,*
\n Pin-Yu Chen3 , \n Ruoxi Jia2 , \n Prateek Mittal1,† ,  \n Peter Henderson4,†  \n
\n1Princeton University   2Virginia Tech   3IBM Research   4Stanford University
\n *Lead Authors    †Equal Advising
\n

\n\n

\n\nICLR (oral), 2024
\n\n

\n

\n\n [arXiv]      [Project Page]      [Dataset]\n\n

\n\n\n------------\n\n$${\\color{red}\\text{\\textbf{!!! Warning !!!}}}$$\n\n$${\\color{red}\\text{\\textbf{This repository contains red-teaming data and }}}$$\n\n$${\\color{red}\\text{\\textbf{model-generated content that can be offensive in nature.}}}$$\n

\n\n**Overview:** Fine-tuning GPT-3.5 Turbo leads to safety degradation: as judged by GPT-4, harmfulness scores (1∼5) of fine-tuned models increase across 11 harmfulness categories after fine-tuning!\n\n\n\nFine-tuning maximizes the likelihood of targets given inputs: \n\n* (a): fine-tuning on 100 explicitly harmful examples; \n* (b): fine-tuning on 10 identity-shifting samples that trick the models into always outputting affirmative prefixes;\n* (c): fine-tuning on the Alpaca dataset.\n\n
\n\n
\n\n## A Quick Glance\n\n\nhttps://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety/assets/146881603/e3b5313d-8ad1-43f1-a561-bdf367277d82\n\n
\n\n
\n\n## On the Safety Risks of Fine-tuning Aligned LLMs\n\n> We evaluate models on a set of harmful instructions we collected. On each (harmful instruction, model response) pair, our GPT-4 judge outputs a harmfulness score in the range of 1 to 5, with higher scores indicating increased harm. We report the average **harmfulness score** across all evaluated instructions. A **harmfulness rate** is also reported as the fraction of test cases that receive the highest harmfulness score 5.\n\n
\n\n### **Risk Level 1**: fine-tuning with explicitly harmful datasets.\n\n\n\n> We jailbreak GPT-3.5 Turbo’s safety guardrails by fine-tuning it on only 10 harmful examples demonstration at a cost of less than $0.20 via OpenAI’s APIs!\n\n\n\n
\n\n### **Risk Level 2**: fine-tuning with implicitly harmful datasets\n\n\n\n> We design a dataset with only [10 manually drafted examples](https://github.com/LLM-Tuning-Safety/LLMs-Finetuning-Safety/blob/main/gpt-3.5/data/identity-shift-aoa.jsonl), none containing explicitly toxic content. These examples aim to","github_created_at":"2023-10-06T16:02:27+00:00","created_at":"2026-07-11T23:40:15.199507+00:00","updated_at":"2026-07-11T23:40:25.869733+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"alignment","name":"alignment"},{"slug":"llm","name":"llm"},{"slug":"llm-finetuning","name":"llm-finetuning"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":701430060,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:40:23.096Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":869,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:40:23.437Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:40:22.822Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:40:22.822Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T23:40:22.822Z"}}}}