{"data":{"slug":"xuandongzhao-weak-to-strong","name":"weak-to-strong","tagline":"Novel Inference-Time Attack Leveraging Small Models to Guide Larger LLMs into Generating Harmful Outputs","github_url":"https://github.com/XuandongZhao/weak-to-strong","owner":"XuandongZhao","repo":"weak-to-strong","owner_avatar_url":"https://avatars.githubusercontent.com/u/26193731?v=4","primary_language":"Python","stars":91,"forks":11,"topics":[],"archived":false,"github_pushed_at":"2025-05-02T02:52:44+00:00","maintenance_label":"Dormant","stars_delta_30d":1,"url":"https://www.graphcanon.com/tools/xuandongzhao-weak-to-strong","markdown_url":"https://www.graphcanon.com/tools/xuandongzhao-weak-to-strong.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xuandongzhao-weak-to-strong","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xuandongzhao-weak-to-strong","description":"[ICML 2025] Weak-to-Strong Jailbreaking on Large Language Models","homepage_url":null,"license":"MIT","open_issues":3,"watchers":3,"ai_summary":"Implements Weak-to-Strong Jailbreaking on Large Language Models (LLMs), which uses small unsafe/aligned models to guide larger aligned models into producing harmful outputs with a high attack success rate.","readme_excerpt":"# Weak-to-Strong Jailbreaking on Large Language Models\n\n📣 **Update**: Our paper has been accepted to **ICML 2025**!  \n\n📄 [arXiv](https://arxiv.org/abs/2401.17256) | 🤗 [HuggingFace Paper Page](https://huggingface.co/papers/2401.17256)\n\n---\n\n## Overview\n\nDespite major advances in aligning large language models (LLMs), red-teaming efforts consistently reveal vulnerabilities: even well-aligned LLMs can be **jailbroken** to produce harmful outputs via adversarial prompts, fine-tuning, or decoding tricks.\n\nThis repository implements **Weak-to-Strong Jailbreaking** — a novel and efficient **inference-time attack** that leverages small (7B) unsafe/aligned LLMs to guide the generation of much larger (e.g., 70B) aligned models into producing unsafe outputs. Surprisingly, the attack only requires **one forward pass through each small model**, making it both **computationally cheap** and **highly effective**.\n\n### Key Insight\n\nAligned and jailbroken LLMs mainly diverge in their **initial decoding steps**. This enables us to apply **log-probability algebra** — using small models to shift the strong model's token distribution early in generation — resulting in **high attack success rates (ASR > 99%)** with **minimal cost**.\n\n---\n\n## Pipeline Illustration\n\n<p align=\"center\">\n  <img src=\"./fig/pipeline.png\" alt=\"pipeline\" width=\"600\"/>\n</p>\n\nWe summarize the trade-offs of different jailbreaking strategies below:\n\n<p align=\"center\">\n  <img src=\"./fig/table.png\" width=\"450\"/>\n</p>\n\n---\n\n## Repository Structure\n\n- `data/`: Contains the data used for the experiments.\n- `run.py`: Contains the scripts used to run the experiments.\n- `generate.py`: Contains the scripts used to generate the results.\n- `eval_asr.py`: Contains the scripts used to evaluate the attack success rate.\n- `eval_gpt.py`: Contains the scripts used to evaluate the GPT4 scores.\n- `eval_harm.py`: Contains the scripts used to evaluate the Harm scores.\n\nFor getting the unsafe small model, please refer to this repo: https://github.com/BeyonderXX/ShadowAlignment\n\n## Running the experiments\n\n```bash\npython run.py --beta 1.50 --batch_size 16 --output_file \"[OUTPUT FILE NAME]\" --att_file \"./data/advbench.txt'\n```\nNeed to confige the bad model path in `run.py` firstly.\n\n## Evaluating the results\n\nFind the examples in `eval_asr.py`, `eval_gpt.py`, and `eval_harm.py` to evaluate the results.\n\n\n## Citation\nIf you find the code useful, please cite the following paper:\n\n```\n@article{zhao2024weak,\n  title={Weak-to-Strong Jailbreaking on Large Language Models},\n  author={Zhao, Xuandong and Yang, Xianjun and Pang, Tianyu and Du, Chao and Li, Lei and Wang, Yu-Xiang and Wang, William Yang},\n  journal={arXiv preprint arXiv:2401.17256},\n  year={2024}\n}\n```","github_created_at":"2024-01-28T19:48:07+00:00","created_at":"2026-07-11T23:39:56.608037+00:00","updated_at":"2026-10-04T06:00:45.955605+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"inference-time-attack","name":"inference-time attack"},{"slug":"jailbreaking","name":"jailbreaking"},{"slug":"large-language-models","name":"large-language-models"}],"trust":{"provenance":{"is_fork":false,"github_id":749514836,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-10-04T06:00:45.025Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":520,"last_release_at":null,"stars_delta_30d":1,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:39:58.088Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-10-04T06:00:45.665Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-10-04T06:00:45.665Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-10-04T06:00:45.665Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["The smaller models guiding the large LLM must be available.","A high-performance computing environment might be necessary if running on very large datasets or models."],"min_ram_gb":8},"constraints":{"min_ram_gb":8},"when_to_use":["Use it for research purposes specifically geared at understanding the vulnerabilities in large language models and improving their robustness against adversarial attacks.","Leverage if your project involves red-teaming efforts that aim to assess how well alignment strategies can be bypassed with minimum computational overhead by manipulating inference steps."],"when_not_to_use":["Do not use it for applications requiring ethical guidelines adherence as it is designed to navigate around the safety mechanisms in large language models.","Avoid using this tool if you are developing systems that must ensure consistent alignment and prevent any form of harmful output generation, such as public communication platforms or education tools."],"source":"enrich:decision_facts","observed_at":"2026-07-16T22:32:21.170Z"},"constraint_facets":{"min_ram_gb":8},"decision_summary":[{"label":"Requirements","value":"Min 8 GB RAM; The smaller models guiding the large LLM must be available.; A high-performance computing environment might be necessary if running on very large datasets or models."},{"label":"Adopt for","value":"Weak-to-Strong is an inference-time attack exploiting smaller models to guide larger LLMs towards harmful output generation."}]}}