{"data":{"slug":"xuandongzhao-weak-to-strong","name":"weak-to-strong","tagline":"[ICML 2025] Weak-to-Strong Jailbreaking on Large Language Models","github_url":"https://github.com/XuandongZhao/weak-to-strong","owner":"XuandongZhao","repo":"weak-to-strong","owner_avatar_url":"https://avatars.githubusercontent.com/u/26193731?v=4","primary_language":"Python","stars":90,"forks":10,"topics":[],"archived":false,"github_pushed_at":"2025-05-02T02:52:44+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/xuandongzhao-weak-to-strong","markdown_url":"https://www.graphcanon.com/tools/xuandongzhao-weak-to-strong.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xuandongzhao-weak-to-strong","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xuandongzhao-weak-to-strong","description":"[ICML 2025] Weak-to-Strong Jailbreaking on Large Language Models","homepage_url":null,"license":"MIT","open_issues":3,"watchers":3,"ai_summary":null,"readme_excerpt":"# Weak-to-Strong Jailbreaking on Large Language Models\n\nšŸ“£ **Update**: Our paper has been accepted to **ICML 2025**! \n\nšŸ“„ [arXiv](https://arxiv.org/abs/2401.17256) | šŸ¤— [HuggingFace Paper Page](https://huggingface.co/papers/2401.17256)\n\n---\n\n## Overview\n\nDespite major advances in aligning large language models (LLMs), red-teaming efforts consistently reveal vulnerabilities: even well-aligned LLMs can be **jailbroken** to produce harmful outputs via adversarial prompts, fine-tuning, or decoding tricks.\n\nThis repository implements **Weak-to-Strong Jailbreaking** ā€” a novel and efficient **inference-time attack** that leverages small (7B) unsafe/aligned LLMs to guide the generation of much larger (e.g., 70B) aligned models into producing unsafe outputs. Surprisingly, the attack only requires **one forward pass through each small model**, making it both **computationally cheap** and **highly effective**.\n\n### Key Insight\n\nAligned and jailbroken LLMs mainly diverge in their **initial decoding steps**. This enables us to apply **log-probability algebra** ā€” using small models to shift the strong model's token distribution early in generation ā€” resulting in **high attack success rates (ASR > 99%)** with **minimal cost**.\n\n---\n\n## Pipeline Illustration\n\n

\n \"pipeline\"\n

\n\nWe summarize the trade-offs of different jailbreaking strategies below:\n\n

\n \n

\n\n---\n\n## Repository Structure\n\n- `data/`: Contains the data used for the experiments.\n- `run.py`: Contains the scripts used to run the experiments.\n- `generate.py`: Contains the scripts used to generate the results.\n- `eval_asr.py`: Contains the scripts used to evaluate the attack success rate.\n- `eval_gpt.py`: Contains the scripts used to evaluate the GPT4 scores.\n- `eval_harm.py`: Contains the scripts used to evaluate the Harm scores.\n\nFor getting the unsafe small model, please refer to this repo: https://github.com/BeyonderXX/ShadowAlignment\n\n## Running the experiments\n\n```bash\npython run.py --beta 1.50 --batch_size 16 --output_file \"[OUTPUT FILE NAME]\" --att_file \"./data/advbench.txt'\n```\nNeed to confige the bad model path in `run.py` firstly.\n\n## Evaluating the results\n\nFind the examples in `eval_asr.py`, `eval_gpt.py`, and `eval_harm.py` to evaluate the results.\n\n\n## Citation\nIf you find the code useful, please cite the following paper:\n\n```\n@article{zhao2024weak,\n title={Weak-to-Strong Jailbreaking on Large Language Models},\n author={Zhao, Xuandong and Yang, Xianjun and Pang, Tianyu and Du, Chao and Li, Lei and Wang, Yu-Xiang and Wang, William Yang},\n journal={arXiv preprint arXiv:2401.17256},\n year={2024}\n}\n```","github_created_at":"2024-01-28T19:48:07+00:00","created_at":"2026-07-11T23:39:56.608037+00:00","updated_at":"2026-07-11T23:39:59.724145+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"speech-audio","name":"Speech & Audio","url":"https://www.graphcanon.com/categories/speech-audio","markdown_url":"https://www.graphcanon.com/categories/speech-audio.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/speech-audio"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":749514836,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:39:57.614Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":435,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:39:58.088Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:39:57.364Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:39:57.364Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T23:39:57.364Z"}}}}