weak-to-strong
Novel Inference-Time Attack Leveraging Small Models to Guide Larger LLMs into Generating Harmful Outputs
GraphCanon updated 2w · GitHub synced 2w
Decision brief
Weak-to-Strong is an inference-time attack exploiting smaller models to guide larger LLMs towards harmful output generation.
Good fit when
- Use it for research purposes specifically geared at understanding the vulnerabilities in large language models and improving their robustness against adversarial attacks.
- Leverage if your project involves red-teaming efforts that aim to assess how well alignment strategies can be bypassed with minimum computational overhead by manipulating inference steps.
Avoid when
- Do not use it for applications requiring ethical guidelines adherence as it is designed to navigate around the safety mechanisms in large language models.
- Avoid using this tool if you are developing systems that must ensure consistent alignment and prevent any form of harmful output generation, such as public communication platforms or education tools.
- Requirements:
- Min 8 GB RAM; The smaller models guiding the large LLM must be available.; A high-performance computing environment might be necessary if running on very large datasets or models.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (459d since push)
- As of 2w
- Provenance
- Not a fork · Personal account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install weak-to-strongSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Implements Weak-to-Strong Jailbreaking on Large Language Models (LLMs), which uses small unsafe/aligned models to guide larger aligned models into producing harmful outputs with a high attack success rate.
Capability facts
- Languages
- python
Source: github.language · Aug 5, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 5, 2026)
python run.py --beta 1.50 --batch_size 16 --output_file "[OUTPUT FILE NAME]" --att_fileSource link
Tags
README
Weak-to-Strong Jailbreaking on Large Language Models
📣 Update: Our paper has been accepted to ICML 2025!
📄 arXiv | 🤗 HuggingFace Paper Page
Overview
Despite major advances in aligning large language models (LLMs), red-teaming efforts consistently reveal vulnerabilities: even well-aligned LLMs can be jailbroken to produce harmful outputs via adversarial prompts, fine-tuning, or decoding tricks.
This repository implements Weak-to-Strong Jailbreaking — a novel and efficient inference-time attack that leverages small (7B) unsafe/aligned LLMs to guide the generation of much larger (e.g., 70B) aligned models into producing unsafe outputs. Surprisingly, the attack only requires one forward pass through each small model, making it both computationally cheap and highly effective.
Key Insight
Aligned and jailbroken LLMs mainly diverge in their initial decoding steps. This enables us to apply log-probability algebra — using small models to shift the strong model's token distribution early in generation — resulting in high attack success rates (ASR > 99%) with minimal cost.
Pipeline Illustration
We summarize the trade-offs of different jailbreaking strategies below:
Repository Structure
data/: Contains the data used for the experiments.run.py: Contains the scripts used to run the experiments.generate.py: Contains the scripts used to generate the results.eval_asr.py: Contains the scripts used to evaluate the attack success rate.eval_gpt.py: Contains the scripts used to evaluate the GPT4 scores.eval_harm.py: Contains the scripts used to evaluate the Harm scores.
For getting the unsafe small model, please refer to this repo: https://github.com/BeyonderXX/ShadowAlignment
Running the experiments
python run.py --beta 1.50 --batch_size 16 --output_file "[OUTPUT FILE NAME]" --att_file "./data/advbench.txt'
Need to confige the bad model path in run.py firstly.
Evaluating the results
Find the examples in eval_asr.py, eval_gpt.py, and eval_harm.py to evaluate the results.
Citation
If you find the code useful, please cite the following paper:
@article{zhao2024weak,
title={Weak-to-Strong Jailbreaking on Large Language Models},
author={Zhao, Xuandong and Yang, Xianjun and Pang, Tianyu and Du, Chao and Li, Lei and Wang, Yu-Xiang and Wang, William Yang},
journal={arXiv preprint arXiv:2401.17256},
year={2024}
}
For agents
This page has a .md twin and JSON over the API.