promptfoo
Tool for evaluating prompts and AI agents by comparing performance across various models and red teaming.
GraphCanon updated 2w · GitHub synced 2w · 25 views this month
Decision brief
promptfoo aids in evaluating AI prompts, LLM agents, and RAG systems through declarative config testing with CI/CD support.
Good fit when
- For comparing performance across GPT, Claude, Gemini, DeepSeek
- When needing to integrate vulnerability scanning within CI/CD pipelines
Avoid when
- If you do not require comparative analysis among multiple LLM models
- If your project does not benefit from the specific red teaming capabilities offered by promptfoo
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Adoption
Package downloads where a registry match exists. GitHub stars (23,838) are secondary evidence.
- npm downloads (30d)
- 1,994,972·npm downloads API·2w
Maintenance and security
Full trust report- Maintenance
- Very active (0d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
npm install promptfooHow it fits your stack(21)
Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.
Alternative
Integrates
Related
Relationship graph
Optional deeper exploration of typed edges and category neighbours.
Similar tools
Same-category neighbours not already linked as typed edges.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Promptfoo enables testing of prompts, LLM-based agents, and RAG systems with support for comparative evaluation among multiple models like GPT, Claude, Gemini, DeepSeek. It offers CI/CD integration and vulnerability scanning capabilities through simple declarative configurations.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 2, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 2, 2026
- CLI
- CLI entrypoint
Source: package.json:bin|scripts · Aug 2, 2026
- MCP server
- No MCP server detected
Source: repo_scan · Aug 2, 2026
- Languages
- typescript, javascript
Source: github.language+package.json · Aug 2, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 2, 2026)
Requires [Node.js](https://nodejs.org/en/download) `>=22.22.0` for npm and npx usage. Node.js 24Source link
Source: README excerpt (regex_v1, Aug 2, 2026)
Also available via `brew install promptfoo` and `pip install promptfoo`. You can also use `npx promptfoo@latest` to run any command withoutSource link
Tags
README
Quick Start
Requires Node.js >=22.22.0 for npm and npx usage. Node.js 24 LTS
is recommended; see the runtime support guide.
npm install -g promptfoo
promptfoo init --example getting-started
Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.
Most LLM providers require an API key. Set yours as an environment variable:
export OPENAI_API_KEY=sk-abc123
Once you're in the example directory, run an eval and view results:
cd getting-started
promptfoo eval
promptfoo view
See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.
For agents
This page has a .md twin and JSON over the API.