--- title: "giskard-oss vs simple-evals" type: "comparison" canonical_url: "https://www.graphcanon.com/compare/giskard-ai-giskard-oss-vs-openai-simple-evals" tools: ["giskard-ai-giskard-oss", "openai-simple-evals"] --- # giskard-oss vs simple-evals *GraphCanon updated Jul 11, 2026* ## Verdict Pick giskard-oss when license: giskard-oss is Apache-2.0, simple-evals is MIT; pick simple-evals when license: simple-evals is MIT, giskard-oss is Apache-2.0. [giskard-oss](https://docs.giskard.ai) reports 5.5k GitHub stars, 485 forks, and 70 open issues, last pushed Jul 10, 2026. [simple-evals](https://github.com/openai/simple-evals) has 4.6k stars, 493 forks, and 56 open issues, last pushed Apr 22, 2026. Figures are from public GitHub metadata via [giskard-oss's repository](https://github.com/Giskard-AI/giskard-oss) and [simple-evals's repository](https://github.com/openai/simple-evals). | | [giskard-oss](/tools/giskard-ai-giskard-oss.md) | [simple-evals](/tools/openai-simple-evals.md) | | --- | --- | --- | | Tagline | 🐢 Open-Source Evaluation & Testing library for LLM Agents | simple-evals | | Stars | 5,505 | 4,565 | | Forks | 485 | 493 | | Open issues | 70 | 56 | | Language | Python | Python | | Adopt for | Giskard-OSS is a Python library aimed at evaluating and testing AI agents, particularly language models. It includes modules for scenario-based tests, security scanning, and synthetic data generation. | - | | Persona | - | - | | Runtime | - | - | | License | Apache-2.0 | MIT | | Categories | Vector Databases, AI Agents, LLM Frameworks | LLM Frameworks, Evaluation & Observability | ## Trust and health _Sourced signals - not a safety guarantee. No winner column._ | | [giskard-oss](/tools/giskard-ai-giskard-oss.md) | [simple-evals](/tools/openai-simple-evals.md) | | --- | --- | --- | | Maintenance | Very active (96%) | Steady (60%) | | Days since push | 1d | 79d | | Open issues (now) | 70 | 56 | | Full report | [trust report](/tools/giskard-ai-giskard-oss/trust.md) | [trust report](/tools/openai-simple-evals/trust.md) | ## Decision facts: giskard-oss - **Requirements:** Requires Python 3.12 or higher. - **Adopt for:** Giskard-OSS is a Python library aimed at evaluating and testing AI agents, particularly language models. It includes modules for scenario-based tests, security scanning, and synthetic data generation. ## Choose when ### Choose giskard-oss if… - License: giskard-oss is Apache-2.0, simple-evals is MIT. - Requirements: Requires Python 3.12 or higher.. - Tags unique to giskard-oss: llm-eval, agent-evaluation, llm, fairness-ai. - Also covers Vector Databases, AI Agents. - - You need an open-source solution specifically designed for testing LLMs with built-in checks and vulnerability scans. ### Choose simple-evals if… - License: simple-evals is MIT, giskard-oss is Apache-2.0. - Tags unique to simple-evals: python. - Also covers Evaluation & Observability. ## When NOT to use giskard-oss - - If you prefer a tool without any potential telemetry data collection, even though Giskard allows opting out, as it could be seen as potentially intrusive despite safeguards. - - You are working with environments that only support Python versions below 3.12, since Giskard-OSS requires at least Python 3.12. ## When NOT to use simple-evals - LLM Frameworks: Avoid a framework for a single prompt-and-retrieve call; the abstraction can cost more than it saves. - Evaluation & Observability: Defer heavyweight eval infra only until you have real traffic - never skip it once users depend on answers. ## Common questions ### What is the difference between giskard-oss and simple-evals? giskard-oss: 🐢 Open-Source Evaluation & Testing library for LLM Agents. simple-evals: simple-evals. See the comparison table for live GitHub stats and shared categories. ### When should I choose giskard-oss over simple-evals? Choose giskard-oss over simple-evals when License: giskard-oss is Apache-2.0, simple-evals is MIT; Requirements: Requires Python 3.12 or higher.; Tags unique to giskard-oss: llm-eval, agent-evaluation, llm, fairness-ai; Also covers Vector Databases, AI Agents; - You need an open-source solution specifically designed for testing LLMs with built-in checks and vulnerability scans. ### When should I choose simple-evals over giskard-oss? Choose simple-evals over giskard-oss when License: simple-evals is MIT, giskard-oss is Apache-2.0; Tags unique to simple-evals: python; Also covers Evaluation & Observability. ### When should I avoid giskard-oss? - If you prefer a tool without any potential telemetry data collection, even though Giskard allows opting out, as it could be seen as potentially intrusive despite safeguards. - You are working with environments that only support Python versions below 3.12, since Giskard-OSS requires at least Python 3.12. ### When should I avoid simple-evals? LLM Frameworks: Avoid a framework for a single prompt-and-retrieve call; the abstraction can cost more than it saves. Evaluation & Observability: Defer heavyweight eval infra only until you have real traffic - never skip it once users depend on answers. ### Is giskard-oss or simple-evals more popular on GitHub? giskard-oss has more GitHub stars (5,505 vs 4,565). Stars measure visibility, not whether either tool fits your constraints. ### Are giskard-oss and simple-evals open source? Yes - both are open-source projects on GitHub (giskard-oss: Apache-2.0, simple-evals: MIT). ### Where can I find alternatives to giskard-oss or simple-evals? GraphCanon lists graph-backed alternatives at [giskard-oss alternatives](/tools/giskard-ai-giskard-oss/alternatives) and [simple-evals alternatives](/tools/openai-simple-evals/alternatives) ([giskard-oss markdown twin](/tools/giskard-ai-giskard-oss/alternatives.md), [simple-evals markdown twin](/tools/openai-simple-evals/alternatives.md)), ranked by typed relationship edges rather than popularity votes. ### Is there a machine-readable version of this comparison? Yes. The markdown twin at [this comparison](/compare/giskard-ai-giskard-oss-vs-openai-simple-evals.md) mirrors this page for agents and LLM crawlers, with the same stats table and FAQ answers. ### Which is better maintained, giskard-oss or simple-evals? giskard-oss: Very active. simple-evals: Steady. Compare maintenance labels, days since push, and release cadence in the trust section below - stars alone do not measure maintenance. ### Where are the full trust reports for giskard-oss and simple-evals? GraphCanon publishes per-repo trust reports with dated maintenance, provenance, and scan summaries: [giskard-oss trust report](/tools/giskard-ai-giskard-oss/trust); [simple-evals trust report](/tools/openai-simple-evals/trust). --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/graph?tool=giskard-ai-giskard-oss`](/api/graphcanon/graph?tool=giskard-ai-giskard-oss) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_