{"data":{"slug":"robustnlp-cipherchat","name":"CipherChat","tagline":"A framework to evaluate the generalization capability of safety alignment for LLMs","github_url":"https://github.com/RobustNLP/CipherChat","owner":"RobustNLP","repo":"CipherChat","owner_avatar_url":"https://avatars.githubusercontent.com/u/71020516?v=4","primary_language":"Python","stars":627,"forks":68,"topics":["alignment","chatgpt","gpt-4-0613","jailbreak","large-language-models","llm","security"],"archived":false,"github_pushed_at":"2025-10-09T03:20:12+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/robustnlp-cipherchat","markdown_url":"https://www.graphcanon.com/tools/robustnlp-cipherchat.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/robustnlp-cipherchat","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=robustnlp-cipherchat","description":"A framework to evaluate the generalization capability of safety alignment for LLMs","homepage_url":null,"license":"MIT","open_issues":0,"watchers":9,"ai_summary":null,"readme_excerpt":"

CipherChat 🔐

\nA novel framework CipherChat to systematically examine the generalizability of safety alignment to non-natural languages – ciphers. \n

\n\nIf you have any questions, please feel free to email the first author: [Youliang Yuan](https://github.com/YouliangYuan).\n \n## 👉 Paper\nFor more details, please refer to our paper [ICLR 2024](https://openreview.net/forum?id=MbfAK4s61A).\n\n\n
\n \"Logo\"\n
\n\n

LOVE💗 and Peace🌊

\n

RESEARCH USE ONLY✅ NO MISUSE❌

\n\n\n## Our results\nWe provide our results (query-response pairs) in `experimental_results`, these files can be loaded by `torch.load()`. Then, you can get a list: the first element is the config and the rest of the elements are the query-response pairs.\n```\nresult_data = torch.load(filename)\nconfig = result_data[0]\npairs = result_data[1:]\n```\n\n\n\n## 🛠️ Usage\n✨An example run:\n```\npython3 main.py \\\n --model_name gpt-4-0613 \\\n--data_path data/data_en_zh.dict \\\n--encode_method caesar \\\n--instruction_type Crimes_And_Illegal_Activities \\\n--demonstration_toxicity toxic \\\n--language en\n```\n## 🔧 Argument Specification\n1. `--model_name`: The name of the model to evaluate.\n\n2. `--data_path`: Select the data to run. \n\n3. `--encode_method`: Select the cipher to use.\n\n4. `--instruction_type`: Select the domain of data.\n\n5. `--demonstration_toxicity`: Select the toxic or safe demonstrations.\n\n6. `--language`: Select the language of the data.\n\n\n## 💡Framework\n
\n \"Logo\"\n
\n\nOur approach presumes that since human feedback and safety alignments are presented in natural language, using a human-unreadable cipher can potentially bypass the safety alignments effectively. Intuitively, we first teach the LLM to comprehend the cipher clearly by designating the LLM as a cipher expert, and elucidating the rules of enciphering and deciphering, supplemented with several demonstrations. We then convert the input into a cipher, which is less likely to be covered by the safety alignment of LLMs, before feeding it to the LLMs. We finally employ a rule-based decrypter to convert the model output from a cipher format into the natural language form. \n\n## 📃Results\nThe query-responses pairs in our experiments are all stored in the form of a list in the \"experimental_results\" folder, and torch.load() can be used to load data.\n
\n \"Logo\"\n
\n\n### 🌰Case Study\n
\n \"Logo\"\n
\n\n### 🫠Ablation Study\n
\n \"Logo\"\n
\n\n### 🦙Other Models\n
\n \"Logo\"\n
\n\n\n\n\n\n\nCommunity Discussion:\n- Twitter: [AIDB](https://twitter.com/ai_database/status/1691655307892830417), [Jiao Wenxiang](https://twitter.com/WenxiangJiao/status/1691363450604457984)\n\n## Citation\n\nIf you find our paper&tool interesting and useful, please feel free to give us a star and cite us through:\n```bibtex\n@inproceedings{yuan2024cipherchat,\n title={GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher},\n author={Yuan, Youliang and Jiao, Wenxiang and Wang, Wenxuan and Huang, Jen-tse and He, Pinjia and Shi, Shuming and Tu, Zhaopeng},\n booktitle={The Twelfth International Conference on Learning Representations}\n}\n```","github_created_at":"2023-08-10T05:55:17+00:00","created_at":"2026-07-11T23:40:07.437001+00:00","updated_at":"2026-07-11T23:40:11.245562+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"alignment","name":"alignment"},{"slug":"chatgpt","name":"chatgpt"},{"slug":"gpt-4-0613","name":"gpt-4-0613"},{"slug":"jailbreak","name":"jailbreak"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"llm","name":"llm"},{"slug":"python","name":"python"},{"slug":"security","name":"security"}],"trust":{"provenance":{"is_fork":false,"github_id":676834494,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:40:09.066Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":275,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:40:09.530Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:40:08.821Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:40:08.821Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T23:40:08.821Z"}}}}