{"data":{"slug":"promptfoo-promptfoo","name":"promptfoo","tagline":"Test prompts, agents, and RAGs. Compare performance of various LLMs.","github_url":"https://github.com/promptfoo/promptfoo","owner":"promptfoo","repo":"promptfoo","owner_avatar_url":"https://avatars.githubusercontent.com/u/137907881?v=4","primary_language":"TypeScript","stars":25251,"forks":2334,"topics":["ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","testing","vulnerability-scanners"],"archived":false,"github_pushed_at":"2026-09-18T07:59:45+00:00","maintenance_label":"Very active","stars_delta_30d":1413,"url":"https://www.graphcanon.com/tools/promptfoo-promptfoo","markdown_url":"https://www.graphcanon.com/tools/promptfoo-promptfoo.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/promptfoo-promptfoo","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=promptfoo-promptfoo","description":"Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration.  Used by OpenAI and Anthropic.","homepage_url":"https://promptfoo.dev","license":"MIT","open_issues":631,"watchers":73,"ai_summary":"A tool for evaluating and testing prompts, AI agents, and retrieval-augmented generation systems. It supports red teaming and vulnerability scanning for AI models, with CI/CD integration.","readme_excerpt":"## Quick Start\n\nRequires [Node.js](https://nodejs.org/en/download) `>=22.22.0` for npm and npx usage. Node.js 24 LTS\nis recommended; see the [runtime support guide](https://www.promptfoo.dev/docs/installation/#nodejs-runtime-support).\n\n```sh\nnpm install -g promptfoo\npromptfoo init --example getting-started\n```\n\nAlso available via `brew install promptfoo` and `pip install promptfoo`. You can also use `npx promptfoo@latest` to run any command without installing.\n\nMost LLM providers require an API key. Set yours as an environment variable:\n\n```sh\nexport OPENAI_API_KEY=sk-abc123\n```\n\nOnce you're in the example directory, run an eval and view results:\n\n```sh\ncd getting-started\npromptfoo eval\npromptfoo view\n```\n\nSee [Getting Started](https://www.promptfoo.dev/docs/getting-started/) (evals) or [Red Teaming](https://www.promptfoo.dev/docs/red-team/) (vulnerability scanning) for more.","github_created_at":"2023-04-28T15:48:49+00:00","created_at":"2026-07-07T17:32:56.693683+00:00","updated_at":"2026-09-20T04:31:03.947698+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"ci","name":"ci"},{"slug":"ci-cd","name":"ci-cd"},{"slug":"cicd","name":"cicd"},{"slug":"evaluation","name":"evaluation"},{"slug":"evaluation-framework","name":"evaluation-framework"},{"slug":"llm-eval","name":"llm-eval"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"llm-evaluation-framework","name":"llm-evaluation-framework"}],"trust":{"provenance":{"is_fork":false,"github_id":633927609,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:28:31.116Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":9,"days_since_push":0,"last_release_at":"2026-09-18T01:07:54Z","stars_delta_30d":1413,"open_issues_delta_30d":150},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:28:33.261Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"mcp":{"source":"repo_scan","observed_at":"2026-09-18T08:28:30.007Z","server_manifest":false},"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:28:30.007Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-09-18T08:28:30.007Z","managed_saas":false},"has_cli":{"value":true,"source":"package.json:bin|scripts","observed_at":"2026-09-18T08:28:30.007Z"},"languages":{"value":["typescript","javascript"],"source":"github.language+package.json","observed_at":"2026-09-18T08:28:30.007Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-09-18T08:28:30.007Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-18T08:28:30.007Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires Node.js >=22.22.0, with Node.js 24 LTS recommended.","API keys for most LLM providers are required and should be set as environment variables."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"requires_docker":false},"when_to_use":["When you need to compare the performance of various LLMs such as GPT, Claude, Gemini, DeepSeek, and more.","If you require a tool that integrates with CI/CD pipelines for automated testing and evaluation.","For red teaming and vulnerability scanning of AI models to ensure robustness and security."],"when_not_to_use":["If you are working with languages other than TypeScript, as Promptfoo is specifically designed for TypeScript environments.","When you do not need to compare multiple LLMs and only require a simple prompt testing tool without advanced features like red teaming."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:06:27.625Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Requires Node.js >=22.22.0, with Node.js 24 LTS recommended.; API keys for most LLM providers are required and should be set as environment variables."},{"label":"Adopt for","value":"Promptfoo is a TypeScript-based tool for evaluating and testing prompts, AI agents, and retrieval-augmented generation systems. It supports red teaming and vulnerability scanning for AI models, with CI/CD integration."}]}}