--- title: "weak-to-strong" type: "tool" slug: "xuandongzhao-weak-to-strong" canonical_url: "https://www.graphcanon.com/tools/xuandongzhao-weak-to-strong" github_url: "https://github.com/XuandongZhao/weak-to-strong" homepage_url: null stars: 90 forks: 10 primary_language: "Python" license: "MIT" archived: false categories: ["llm-frameworks", "inference-serving", "speech-audio"] tags: ["python"] updated_at: "2026-07-11T23:39:59.724145+00:00" --- # weak-to-strong > [ICML 2025] Weak-to-Strong Jailbreaking on Large Language Models [ICML 2025] Weak-to-Strong Jailbreaking on Large Language Models ## Facts - Repository: https://github.com/XuandongZhao/weak-to-strong - Stars: 90 · Forks: 10 · Open issues: 3 · Watchers: 3 - Primary language: Python - License: MIT - Last pushed: 2025-05-02T02:52:44+00:00 ## Trust & health _Signals computed from public GitHub metadata. Not a security guarantee._ - Maintenance: Dormant (computed 2026-07-11T23:39:57.614Z) - Security scan: No lockfile (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-11T23:39:58.088Z - Full report: [trust report](/tools/xuandongzhao-weak-to-strong/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/xuandongzhao-weak-to-strong/trust) ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) - [Inference & Serving](/categories/inference-serving.md) - [Speech & Audio](/categories/speech-audio.md) ## Tags python ## Category neighbours (exploratory) _Same-category tools for discovery only - not curated alternatives. Cap shown at six._ - [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active] - [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active] - [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active] - [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active] - [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 162,482) [Very active] - [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful tool for building and deploying AI-powered agents and workflows. (★ 151,697) [Very active] _+ 2 more not listed._ ## README (excerpt) _Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._ ````text # Weak-to-Strong Jailbreaking on Large Language Models 📣 **Update**: Our paper has been accepted to **ICML 2025**! 📄 [arXiv](https://arxiv.org/abs/2401.17256) | 🤗 [HuggingFace Paper Page](https://huggingface.co/papers/2401.17256) --- ## Overview Despite major advances in aligning large language models (LLMs), red-teaming efforts consistently reveal vulnerabilities: even well-aligned LLMs can be **jailbroken** to produce harmful outputs via adversarial prompts, fine-tuning, or decoding tricks. This repository implements **Weak-to-Strong Jailbreaking** — a novel and efficient **inference-time attack** that leverages small (7B) unsafe/aligned LLMs to guide the generation of much larger (e.g., 70B) aligned models into producing unsafe outputs. Surprisingly, the attack only requires **one forward pass through each small model**, making it both **computationally cheap** and **highly effective**. ### Key Insight Aligned and jailbroken LLMs mainly diverge in their **initial decoding steps**. This enables us to apply **log-probability algebra** — using small models to shift the strong model's token distribution early in generation — resulting in **high attack success rates (ASR > 99%)** with **minimal cost**. --- ## Pipeline Illustration

pipeline

We summarize the trade-offs of different jailbreaking strategies below:

--- ## Repository Structure - `data/`: Contains the data used for the experiments. - `run.py`: Contains the scripts used to run the experiments. - `generate.py`: Contains the scripts used to generate the results. - `eval_asr.py`: Contains the scripts used to evaluate the attack success rate. - `eval_gpt.py`: Contains the scripts used to evaluate the GPT4 scores. - `eval_harm.py`: Contains the scripts used to evaluate the Harm scores. For getting the unsafe small model, please refer to this repo: https://github.com/BeyonderXX/ShadowAlignment ## Running the experiments ```bash python run.py --beta 1.50 --batch_size 16 --output_file "[OUTPUT FILE NAME]" --att_file "./data/advbench.txt' ``` Need to confige the bad model path in `run.py` firstly. ## Evaluating the results Find the examples in `eval_asr.py`, `eval_gpt.py`, and `eval_harm.py` to evaluate the results. ## Citation If you find the code useful, please cite the following paper: ``` @article{zhao2024weak, title={Weak-to-Strong Jailbreaking on Large Language Models}, author={Zhao, Xuandong and Yang, Xianjun and Pang, Tianyu and Du, Chao and Li, Lei and Wang, Yu-Xiang and Wang, William Yang}, journal={arXiv preprint arXiv:2401.17256}, year={2024} } ``` ```` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/xuandongzhao-weak-to-strong`](/api/graphcanon/tools/xuandongzhao-weak-to-strong) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_