{"data":{"slug":"neelsjain-baseline-defenses","name":"baseline-defenses","tagline":"Research code for evaluating defenses against adversarial attacks on aligned language models","github_url":"https://github.com/neelsjain/baseline-defenses","owner":"neelsjain","repo":"baseline-defenses","owner_avatar_url":"https://avatars.githubusercontent.com/u/90730659?v=4","primary_language":"Python","stars":35,"forks":1,"topics":[],"archived":false,"github_pushed_at":"2023-10-26T22:00:06+00:00","maintenance_label":"Dormant","stars_delta_30d":1,"url":"https://www.graphcanon.com/tools/neelsjain-baseline-defenses","markdown_url":"https://www.graphcanon.com/tools/neelsjain-baseline-defenses.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/neelsjain-baseline-defenses","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=neelsjain-baseline-defenses","description":"Official Code for \"Baseline Defenses for Adversarial Attacks Against Aligned Language Models\"","homepage_url":null,"license":null,"open_issues":0,"watchers":1,"ai_summary":"Includes code for perplexity filter and paraphrase attack defense strategies. Evaluation of detection, input preprocessing, and adversarial training methods.","readme_excerpt":"# Baseline Defenses for Adversarial Attacks Against Aligned Language Models\nOfficial Code for \"Baseline Defenses for Adversarial Attacks Against Aligned Language Models\"\n\n## Overview\nWe evaluate several baseline defense strategies against leading adversarial attacks on LLMs, discussing the various settings in which each is feasible and effective. Particularly, we look at three types of defenses: detection (perplexity based), input preprocessing (paraphrase and retokenization), and adversarial training. The paper can be found [here](https://arxiv.org/abs/2309.00614).\n\nThe repository only contains the code for the perplexity filter and paraphrase attack. The retokenization defenses is conducted directly via altering tokenizer via BPE-dropout. For LLaMA model, see the `tokenizer.sp_model.encode(input_text, alpha=bt_alpha, enable_sampling=True)` function, and for other models, BPE-dropout is set by `tokenizer._tokenizer.model.dropout=bt_alpha`, where `bt_alpha` is the dropout rate.\n\n## Perplexity Filter\n\nThe perplexity filter in the code consists of two filters, a perplexity filter which as also been proposed in concurrent work by [Alon et al.](https://arxiv.org/abs/2308.14132) and a windowed perplexity filter, which consists of checking the perplexity of a window of $n$ tokens.\n\n## Paraphrase Defense\n\nThe paraphrase defense is rewriting the prompt. For our experiments, we used ChatGPT. Note while this defense is effective it might come at high performance cost.\n\n## Limitations\nAs in all research work, we were limited to the settings we explored in the paper.","github_created_at":"2023-10-26T21:41:28+00:00","created_at":"2026-07-11T23:41:25.359046+00:00","updated_at":"2026-10-05T06:00:45.65209+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"adversarial-attacks","name":"adversarial-attacks"},{"slug":"defense-strategies","name":"defense strategies"},{"slug":"paraphrase-defense","name":"paraphrase defense"},{"slug":"perplexity-filter","name":"perplexity filter"}],"trust":{"provenance":{"is_fork":false,"github_id":710521062,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-10-05T06:00:44.589Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1074,"last_release_at":null,"stars_delta_30d":1,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:41:28.765Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-10-05T06:00:45.338Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-10-05T06:00:45.338Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["- When you need to evaluate the effectiveness of baseline defenses such as the perplexity filter or paraphrase defense in protecting aligned language models from adversarial attacks.","- If your work involves examining how input preprocessing, like paraphrasing prompts, can mitigate risks associated with adversarial inputs."],"when_not_to_use":["- Do not use if you require comprehensive coverage of all possible defensive measures. This tool specifically lacks detailed code for retokenization defenses involving BPE-dropout.","- If your scenario demands more advanced or specialized defense mechanisms beyond the scope of baseline strategies, this repository will fall short on delivering those."],"source":"enrich:decision_facts","observed_at":"2026-07-12T13:13:28.220Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"A toolkit for evaluating defenses against adversarial attacks on aligned language models, focusing on perplexity filter and paraphrase defense strategies."}]}}