---
title: "CipherChat"
type: "tool"
slug: "robustnlp-cipherchat"
canonical_url: "https://www.graphcanon.com/tools/robustnlp-cipherchat"
github_url: "https://github.com/RobustNLP/CipherChat"
homepage_url: null
stars: 627
forks: 68
primary_language: "Python"
license: "MIT"
archived: false
categories: ["evaluation-observability", "llm-frameworks"]
tags: ["alignment", "chatgpt", "gpt-4-0613", "jailbreak", "large-language-models", "llm", "python", "security"]
updated_at: "2026-07-11T23:40:11.245562+00:00"
---
# CipherChat
> A framework to evaluate the generalization capability of safety alignment for LLMs
A framework to evaluate the generalization capability of safety alignment for LLMs
## Facts
- Repository: https://github.com/RobustNLP/CipherChat
- Stars: 627 · Forks: 68 · Open issues: 0 · Watchers: 9
- Primary language: Python
- License: MIT
- Last pushed: 2025-10-09T03:20:12+00:00
## Trust & health
_Signals computed from public GitHub metadata. Not a security guarantee._
- Maintenance: Slowing (computed 2026-07-11T23:40:09.066Z)
- Security scan: No lockfile (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-11T23:40:09.530Z
- Full report: [trust report](/tools/robustnlp-cipherchat/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/robustnlp-cipherchat/trust)
## Categories
- [Evaluation & Observability](/categories/evaluation-observability.md)
- [LLM Frameworks](/categories/llm-frameworks.md)
## Tags
alignment, chatgpt, gpt-4-0613, jailbreak, large-language-models, llm, python, security
## Category neighbours (exploratory)
_Same-category tools for discovery only - not curated alternatives. Cap shown at six._
- [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active]
- [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 162,482) [Very active]
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 145,029) [Very active]
_+ 2 more not listed._
## README (excerpt)
_Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._
````text
CipherChat 🔐
A novel framework CipherChat to systematically examine the generalizability of safety alignment to non-natural languages – ciphers.
If you have any questions, please feel free to email the first author: [Youliang Yuan](https://github.com/YouliangYuan).
## 👉 Paper
For more details, please refer to our paper [ICLR 2024](https://openreview.net/forum?id=MbfAK4s61A).
LOVE💗 and Peace🌊
RESEARCH USE ONLY✅ NO MISUSE❌
## Our results
We provide our results (query-response pairs) in `experimental_results`, these files can be loaded by `torch.load()`. Then, you can get a list: the first element is the config and the rest of the elements are the query-response pairs.
```
result_data = torch.load(filename)
config = result_data[0]
pairs = result_data[1:]
```
## 🛠️ Usage
✨An example run:
```
python3 main.py \
--model_name gpt-4-0613 \
--data_path data/data_en_zh.dict \
--encode_method caesar \
--instruction_type Crimes_And_Illegal_Activities \
--demonstration_toxicity toxic \
--language en
```
## 🔧 Argument Specification
1. `--model_name`: The name of the model to evaluate.
2. `--data_path`: Select the data to run.
3. `--encode_method`: Select the cipher to use.
4. `--instruction_type`: Select the domain of data.
5. `--demonstration_toxicity`: Select the toxic or safe demonstrations.
6. `--language`: Select the language of the data.
## 💡Framework
Our approach presumes that since human feedback and safety alignments are presented in natural language, using a human-unreadable cipher can potentially bypass the safety alignments effectively. Intuitively, we first teach the LLM to comprehend the cipher clearly by designating the LLM as a cipher expert, and elucidating the rules of enciphering and deciphering, supplemented with several demonstrations. We then convert the input into a cipher, which is less likely to be covered by the safety alignment of LLMs, before feeding it to the LLMs. We finally employ a rule-based decrypter to convert the model output from a cipher format into the natural language form.
## 📃Results
The query-responses pairs in our experiments are all stored in the form of a list in the "experimental_results" folder, and torch.load() can be used to load data.
### 🌰Case Study
### 🫠Ablation Study
### 🦙Other Models
Community Discussion:
- Twitter: [AIDB](https://twitter.com/ai_database/status/1691655307892830417), [Jiao Wenxiang](https://twitter.com/WenxiangJiao/status/1691363450604457984)
## Citation
If you find our paper&tool interesting and useful, please feel free to give us a star and cite us through:
```bibtex
@inproceedings{yuan2024cipherchat,
title={GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher},
author={Yuan, Youliang and Jiao, Wenxiang and Wang, Wenxuan and Huang, Jen-tse and He, Pinjia and Shi, Shuming and Tu, Zhaopeng},
booktitle={The Twelfth International Conference on Learning Representations}
}
```
````
---
**Machine-readable endpoints**
- JSON: [`/api/graphcanon/tools/robustnlp-cipherchat`](/api/graphcanon/tools/robustnlp-cipherchat)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)
_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_
