---
title: "promptfoo"
type: "tool"
slug: "promptfoo-promptfoo"
canonical_url: "https://www.graphcanon.com/tools/promptfoo-promptfoo"
github_url: "https://github.com/promptfoo/promptfoo"
homepage_url: "https://promptfoo.dev"
stars: 23003
forks: 2054
primary_language: "TypeScript"
license: "MIT"
categories: ["evaluation-observability", "developer-tools"]
tags: ["vulnerability-scanners", "red-teaming", "prompt-engineering", "ci", "cicd", "evaluation-framework", "pentesting", "llm-evaluation"]
updated_at: "2026-07-07T18:25:20.01904+00:00"
---

# promptfoo

> Test your prompts, agents, and RAGs. Evaluate LLM performance with red teaming/pentesting/vulnerability scanning.

A comprehensive evaluation framework for large language models (LLMs) providing tools to test prompts, perform red teaming exercises, compare multiple LLM performances, and integrate with CI/CD pipelines using simple declarative configurations.

## Facts

- Repository: https://github.com/promptfoo/promptfoo
- Homepage: https://promptfoo.dev
- Stars: 23,003 · Forks: 2,054 · Open issues: 403 · Watchers: 58
- Primary language: TypeScript
- License: MIT
- Last pushed: 2026-07-07T15:23:00+00:00

## Categories

- [Evaluation & Observability](/categories/evaluation-observability.md)
- [Developer Tools](/categories/developer-tools.md)

## Tags

vulnerability-scanners, red-teaming, prompt-engineering, ci, cicd, evaluation-framework, pentesting, llm-evaluation

## Related tools

- [ECC](/tools/affaan-m-ecc.md) - The agent harness performance optimization system (★ 226,962)
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT: Build, Deploy, and Run AI Agents (★ 185,417)
- [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019)
- [JavaGuide](/tools/snailclimb-javaguide.md) - Snailclimb/JavaGuide: 面试 & 后端通用面试指南，覆盖计算机基础、数据库、分布式、高并发、系统设计与 AI 应用开发 (★ 156,863)
- [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful platform for building and deploying AI-powered agents and workflows. (★ 151,298)
- [browser-use](/tools/browser-use-browser-use.md) - 🌐 Make websites accessible for AI agents. Automate tasks online with ease. (★ 103,315)
- [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,150)
- [llm-course](/tools/mlabonne-llm-course.md) - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks. (★ 80,726)

## README (excerpt)

```text
# Promptfoo: LLM evals & red teaming

<p align="center">
  <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/v/promptfoo" alt="npm"></a>
  <a href="https://npmjs.com/package/promptfoo"><img src="https://img.shields.io/npm/dm/promptfoo" alt="npm"></a>
  <a href="https://github.com/promptfoo/promptfoo/actions/workflows/main.yml"><img src="https://img.shields.io/github/actions/workflow/status/promptfoo/promptfoo/main.yml" alt="GitHub Workflow Status"></a>
  <a href="https://github.com/promptfoo/promptfoo/blob/main/LICENSE"><img src="https://img.shields.io/github/license/promptfoo/promptfoo" alt="MIT license"></a>
  <a href="https://discord.gg/promptfoo"><img src="https://img.shields.io/discord/1146610656779440188?logo=discord&label=promptfoo" alt="Discord"></a>
</p>

<p align="center">
  <code>promptfoo</code> is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps.
</p>

<p align="center">
  <a href="https://www.promptfoo.dev">Website</a> ·
  <a href="https://www.promptfoo.dev/docs/getting-started/">Getting Started</a> ·
  <a href="https://www.promptfoo.dev/docs/red-team/">Red Teaming</a> ·
  <a href="https://www.promptfoo.dev/docs/">Documentation</a> ·
  <a href="https://discord.gg/promptfoo">Discord</a>
</p>

> Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the [company update](https://www.promptfoo.dev/blog/promptfoo-joining-openai/).

## Quick Start

Requires [Node.js](https://nodejs.org/en/download) `^20.20.0` or `>=22.22.0` for npm and npx usage.
[Node.js 20 support ends July 30, 2026 at 00:00 UTC](https://www.promptfoo.dev/docs/installation/#nodejs-runtime-support); upgrade to Node.js 24 LTS before updating promptfoo at or after the cutoff.

```sh
npm install -g promptfoo
promptfoo init --example getting-started
```

Also available via `brew install promptfoo` and `pip install promptfoo`. You can also use `npx promptfoo@latest` to run any command without installing.

Most LLM providers require an API key. Set yours as an environment variable:

```sh
export OPENAI_API_KEY=sk-abc123
```

Once you're in the example directory, run an eval and view results:

```sh
cd getting-started
promptfoo eval
promptfoo view
```

See [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.

## What can you do with Promptfoo?

- **Test your prompts and models** with [automated evaluations](https://www.promptfoo.dev/docs/getting-started/)
- **Secure your LLM apps** with [red teaming](https://www.promptfoo.dev/docs/red-team/) and vulnerability scanning
- **Compare models** side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and [more](https://www.promptfoo.dev/docs/providers/))
- **Automate checks** in [CI/CD](https://www.promptfoo.dev/docs/integrations/ci-cd/)
- **Review pull requests** for LLM-related security and compliance issues with [code scanning](https://www.promptfoo.dev/docs/code-scanning/)
- **Share results** with your team

Here's what it looks like in action:

<img src="site/static/img/claude-vs-gpt-example@2x.png" alt="prompt evaluation matrix - web viewer" width="700">

It works on the command line too:

<img src="https://www.promptfoo.dev/img/docs/self-grading.gif" alt="promptfoo command line" width="700">

It also can generate [security vulnerability reports](https://www.promptfoo.dev/docs/red-team/):

<img src="https://www.promptfoo.dev/img/redteam-dashboard@2x.jpg" alt="gen ai red team" width="700">

## Why Promptfoo?

- **Developer-first**: Fast, with features like live reload and caching
- **Private**: LLM evals run 100% locally - your prompts never leave your machine
- **Flexible**: Works with any LLM API or programming language
- **Battle-tested**: Powers LLM apps serving 10M+ users in production
- **Data-driven**: Make decisions based on metrics, not gu
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/promptfoo-promptfoo`](/api/graphcanon/tools/promptfoo-promptfoo)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_