---
title: "ALERT"
type: "tool"
slug: "babelscape-alert"
canonical_url: "https://www.graphcanon.com/tools/babelscape-alert"
github_url: "https://github.com/Babelscape/ALERT"
homepage_url: "https://arxiv.org/abs/2404.08676"
stars: 60
forks: 9
primary_language: "Python"
license: "Other"
archived: false
categories: ["computer-vision", "llm-frameworks", "model-training"]
tags: ["ai", "artificial-intelligence", "benchmark", "bias-detection", "llm", "llm-evaluation", "llm-safety", "llm-safety-benchmark"]
updated_at: "2026-07-15T10:40:28.861975+00:00"
---

# ALERT

> Official repository for the paper "ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming"

Official repository for the paper "ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming"

## Facts

- Repository: https://github.com/Babelscape/ALERT
- Homepage: https://arxiv.org/abs/2404.08676
- Stars: 60 · Forks: 9 · Open issues: 0 · Watchers: 2
- Primary language: Python
- License: Other
- Last pushed: 2024-09-20T08:29:57+00:00

## Trust & health

_Signals computed from public GitHub metadata. Not a security guarantee._

- Maintenance: Dormant (computed 2026-07-15T10:40:27.156Z)
- Security scan: No findings reported (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-15T10:40:27.496Z
- Full report: [trust report](/tools/babelscape-alert/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/babelscape-alert/trust)

## Categories

- [Computer Vision](/categories/computer-vision.md)
- [LLM Frameworks](/categories/llm-frameworks.md)
- [Model Training](/categories/model-training.md)

## Tags

ai, artificial-intelligence, benchmark, bias-detection, llm, llm-evaluation, llm-safety, llm-safety-benchmark

## Category neighbours (exploratory)

_Same-category tools for discovery only - not curated alternatives. Cap shown at six._

- [tensorflow](/tools/tensorflow-tensorflow.md) - An Open Source Machine Learning Framework for Everyone (★ 196,300) [Very active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active]
- [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 162,482) [Very active]
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 145,029) [Very active]

_+ 2 more not listed._

## README (excerpt)

_Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._

```text
# License
As specified in the paper, most of the prompts available in the ALERT benchmark are derived from the [Anthropic HH-RLHF dataset](https://github.com/anthropics/hh-rlhf/tree/master?tab=readme-ov-file) that is licensed under the MIT license. A copy of the license can be found [here](https://github.com/Babelscape/ALERT/blob/master/MIT_LICENSE). 

Starting from these prompts, we then employ a combination of keyword-matching and zero-shot classification strategies to filter out prompts that do not target one of our safety risk categories as well as to classify remaining ones. Furthermore, we designed templates to create new, additional prompts and provide sufficient support for each safety risk category in our benchmark. Finally, we adopt adversarial data augmentation methods to create the ALERT<sub>Adv</sub> subset of our benchmark. The ALERT benchmark is licensed under the CC BY-NC-SA 4.0 license. The text of the license can be found [here](https://github.com/Babelscape/ALERT/blob/master/LICENSE).

<br>
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/babelscape-alert`](/api/graphcanon/tools/babelscape-alert)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_