{"data":{"slug":"promptfoo-promptfoo","name":"promptfoo","tagline":"Tool for evaluating prompts and AI agents by comparing performance across various models and red teaming.","github_url":"https://github.com/promptfoo/promptfoo","owner":"promptfoo","repo":"promptfoo","owner_avatar_url":"https://avatars.githubusercontent.com/u/137907881?v=4","primary_language":"TypeScript","stars":23838,"forks":2147,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"archived":false,"github_pushed_at":"2026-08-01T23:47:56+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/promptfoo-promptfoo","markdown_url":"https://www.graphcanon.com/tools/promptfoo-promptfoo.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/promptfoo-promptfoo","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=promptfoo-promptfoo","description":"Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration.  Used by OpenAI and Anthropic.","homepage_url":"https://promptfoo.dev","license":"MIT","open_issues":481,"watchers":61,"ai_summary":"Promptfoo enables testing of prompts, LLM-based agents, and RAG systems with support for comparative evaluation among multiple models like GPT, Claude, Gemini, DeepSeek. It offers CI/CD integration and vulnerability scanning capabilities through simple declarative configurations.","readme_excerpt":"## Quick Start\n\nRequires [Node.js](https://nodejs.org/en/download) `>=22.22.0` for npm and npx usage. Node.js 24 LTS\nis recommended; see the [runtime support guide](https://www.promptfoo.dev/docs/installation/#nodejs-runtime-support).\n\n```sh\nnpm install -g promptfoo\npromptfoo init --example getting-started\n```\n\nAlso available via `brew install promptfoo` and `pip install promptfoo`. You can also use `npx promptfoo@latest` to run any command without installing.\n\nMost LLM providers require an API key. Set yours as an environment variable:\n\n```sh\nexport OPENAI_API_KEY=sk-abc123\n```\n\nOnce you're in the example directory, run an eval and view results:\n\n```sh\ncd getting-started\npromptfoo eval\npromptfoo view\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.","github_created_at":"2023-04-28T15:48:49+00:00","created_at":"2026-07-07T17:32:56.693683+00:00","updated_at":"2026-08-02T12:00:46.80407+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"ci-cd","name":"ci-cd"},{"slug":"evaluation-framework","name":"evaluation-framework"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"pentesting","name":"pentesting"},{"slug":"red-teaming","name":"red-teaming"},{"slug":"vulnerability-scanners","name":"vulnerability-scanners"}],"trust":{"provenance":{"is_fork":false,"github_id":633927609,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-02T12:00:45.741Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":14,"days_since_push":0,"last_release_at":"2026-07-31T03:54:07Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:17:52.761Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"mcp":{"source":"repo_scan","observed_at":"2026-08-02T12:00:46.178Z","server_manifest":false},"scan":{"source":"repo_scan","observed_at":"2026-08-02T12:00:46.178Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-02T12:00:46.178Z","managed_saas":false},"has_cli":{"value":true,"source":"package.json:bin|scripts","observed_at":"2026-08-02T12:00:46.178Z"},"languages":{"value":["typescript","javascript"],"source":"github.language+package.json","observed_at":"2026-08-02T12:00:46.178Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-02T12:00:46.178Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-02T12:00:46.178Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["For comparing performance across GPT, Claude, Gemini, DeepSeek","When needing to integrate vulnerability scanning within CI/CD pipelines"],"when_not_to_use":["If you do not require comparative analysis among multiple LLM models","If your project does not benefit from the specific red teaming capabilities offered by promptfoo"],"source":"enrich:decision_facts","observed_at":"2026-07-17T04:13:27.221Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"promptfoo aids in evaluating AI prompts, LLM agents, and RAG systems through declarative config testing with CI/CD support."}]}}