Awesome LLM Security

A curation of awesome tools, documents and projects about LLM Security.

Contributions are always welcome. Please read the Contribution Guidelines before contributing.

[!NOTE] ⚡ For efficient research navigation, we’re sharing PDFs via Moonlight, which provides summaries alongside the original paper.

Table of Contents

• Awesome LLM Security
  • Table of Contents
  • Papers
    • White-box attack
    • Black-box attack
    • Backdoor attack
    • Fingerprinting
    • Defense
    • Platform Security
    • Survey
  • Benchmark
  • Tools
  • Articles
  • Other Awesome Projects
  • Other Useful Resources

Papers

White-box attack

• "Visual Adversarial Examples Jailbreak Large Language Models", 2023-06, AAAI(Oral) 24, multi-modal, [paper] [repo]
• "Are aligned neural networks adversarially aligned?", 2023-06, NeurIPS(Poster) 23, multi-modal, [paper]
• "(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs", 2023-07, multi-modal [paper]
• "Universal and Transferable Adversarial Attacks on Aligned Language Models", 2023-07, transfer, [paper] [repo] [page]
• "Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models", 2023-07, multi-modal, [paper]
• "Image Hijacking: Adversarial Images can Control Generative Models at Runtime", 2023-09, multi-modal, [paper] [repo] [site]
• "Weak-to-Strong Jailbreaking on Large Language Models", 2024-04, token-prob, [paper] [repo]

Black-box attack

• "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", 2023-02, AISec@CCS 23 [paper]
• "Jailbroken: How Does LLM Safety Training Fail?", 2023-07, NeurIPS(Oral) 23, [paper]
• "Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models", 2023-07, [paper] [repo]
• "Effective Prompt Extraction from Language Models", 2023-07, prompt-extraction, [paper]
• "Multi-step Jailbreaking Privacy Attacks on ChatGPT", 2023-04, EMNLP 23, privacy, [paper]
• "LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?", 2023-07, [paper]
• "Jailbreaking chatgpt via prompt engineering: An empirical study", 2023-05, [[paper]](https://www.themoonligh