Home/Evaluation & Observability/promptfoo

promptfoo

promptfoo/promptfoo

Test prompts, agents, and RAGs. Compare performance of various LLMs.

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

123views this month

25k stars2.3k forksLast push Sep 18, 2026 TypeScript MIT

Decision brief

Promptfoo is a TypeScript-based tool for evaluating and testing prompts, AI agents, and retrieval-augmented generation systems. It supports red teaming and vulnerability scanning for AI models, with CI/CD integration.

Good fit when

When you need to compare the performance of various LLMs such as GPT, Claude, Gemini, DeepSeek, and more.
If you require a tool that integrates with CI/CD pipelines for automated testing and evaluation.

Avoid when

If you are working with languages other than TypeScript, as Promptfoo is specifically designed for TypeScript environments.
When you do not need to compare multiple LLMs and only require a simple prompt testing tool without advanced features like red teaming.

Requirements:: Requires Node.js >=22.22.0, with Node.js 24 LTS recommended.; API keys for most LLM providers are required and should be set as environment variables.

Observed Sep 18, 2026 · Source: enrich:decision_facts

promptfoo alternatives Trust report

promptfoo vs giskard-oss promptfoo vs langfuse promptfoo vs evidently

Verify the decision

Adoption

Package downloads where a registry match exists. GitHub stars (25,251) are secondary evidence.

npm downloads (30d): 2,633,102·npm downloads API·Sep 18, 2026

Maintenance and security

Full trust report

Maintenance: Very active (0d since push); As of Sep 18, 2026
Provenance: Not a fork · Organization account; As of Sep 18, 2026
Security (OSV): No lockfile; As of Sep 18, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

npm install promptfoo

npm

How it fits your stack(21)

Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.

Alternative

prompt-optimizer

Both Prompt Optimizer and promptfoo are tools focused on evaluating and enhancing the effectiveness of prompts used with LLMs, albeit through different methodologies.

alternative

langfuse

Both Promptfoo and Langfuse offer evaluation and observability for LLMs, but they do so through different methodologies and toolsets.

alternative

opik

Both Promptfoo and OPiK provide tools for evaluating, testing, and improving the performance of LLM models and systems.

alternative

Integrates

langgraph

Promptfoo integrates with Langgraph to allow for the evaluation and testing of stateful agents built using LangGraph's framework, enabling comprehensive performance assessment and security analysis of these complex systems.

integrates

hexstrike-ai

HexStrike AI MCP Agents could benefit from integrating PromptFoo for evaluating and red-teaming the LLM apps that are part of its autonomous AI agents. This ensures robustness in their execution.

integrates

langwatch

Both LangWatch and PromptFoo focus on evaluating and testing language models and their prompts, making them complementary for comprehensive LLM evaluation.

integrates

Prompt-Engineering-Guide

Guides, papers, lessons, notebooks and resources for prompt engineering, context engineering, RAG, and AI Agents

gitleaks

Both Gitleaks and Promptfoo focus on data security within AI applications, albeit in different capacities.

RagaAI-Catalyst

Both Promptfoo and RagaAI-Catalyst aim to enhance monitoring, evaluation, and observability for LLMs and AI agents, but from different angles.

Relationship graph

Optional deeper exploration of typed edges and category neighbours.

Similar tools

Same-category neighbours not already linked as typed edges.

system_prompts_leaks

Leaked system prompts from various AI agents

67k★

ai-engineering-hub

Tutorials on LLMs, RAGs, and real-world AI agent applications

37k★

awesome-ai-apps

A curated list of AI applications showcasing RAG, agents, and workflows.

16k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

A tool for evaluating and testing prompts, AI agents, and retrieval-augmented generation systems. It supports red teaming and vulnerability scanning for AI models, with CI/CD integration.

Capability facts

Deploy: Self-host
Source: dockerfile:Dockerfile · Sep 18, 2026
Docker: Dockerfile present
Source: dockerfile:Dockerfile · Sep 18, 2026
CLI: CLI entrypoint
Source: package.json:bin|scripts · Sep 18, 2026
MCP server: No MCP server detected
Source: repo_scan · Sep 18, 2026
Languages: typescript, javascript
Source: github.language+package.json · Sep 18, 2026

Graph entities

Ollamaframework

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Node.js runtimeNode.js

Source: README excerpt (regex_v1, Sep 18, 2026)

Requires [Node.js](https://nodejs.org/en/download) `>=22.22.0` for npm and npx usage. Node.js 24

Source link

Python runtimePython

Source: README excerpt (regex_v1, Sep 18, 2026)

Also available via `brew install promptfoo` and `pip install promptfoo`. You can also use `npx promptfoo@latest` to run any command without

Source link

README

Quick Start Requires Node.js for npm and npx usage. Node.js 24 LTS is recommended; see the runtime support guide. Also available via and . You can also use to run any command without installing. Most LLM providers require an API key. Set yours as an environment variable: Once you...

For agents

This page has a .md twin and JSON over the API.

/tools/promptfoo-promptfoo.md GET /api/graphcanon/tools/promptfoo-promptfoo GET /api/graphcanon/graph?tool=promptfoo-promptfoo

promptfoo

Decision brief

Good fit when

Avoid when

Verify the decision

Adoption

Maintenance and security

Install

How it fits your stack(21)

Alternative

Integrates

Related

Relationship graph

Similar tools

Evidence and technical details

Overview

Capability facts

Categories

Graph entities

Compatibility

Tags

README

For agents