Evals, Red Teaming and Test Generation for Agentic Systems

Modular, Lightweight, Dynamic and Async-first

Docs • Website • Community

[!IMPORTANT] Giskard v3 is a fresh rewrite designed for dynamic, multi-turn testing of AI agents. This release drops heavy dependencies for better efficiency while introducing a more powerful AI vulnerability scanner and enhanced RAG evaluation capabilities. For now, the vulnerability scanner and RAG evaluation still rely on Giskard v2. Giskard v2 remains available but is no longer actively maintained. Follow progress → Read the v3 Announcement · Roadmap

Install

pip install giskard

Requires Python 3.12+.

Telemetry: Libraries built on giskard-core (including giskard-checks) may send optional, aggregated usage analytics to help improve the product. No prompts, model outputs, or scenario text are included. See what is collected and how to opt out.

---

Giskard is an open-source Python library for testing and evaluating agentic systems. The v3 architecture is a modular set of focused packages — each carrying only the dependencies it needs — built from scratch to wrap anything: an LLM, a black-box agent, or a multi-step pipeline.

Status	Package	Description
✅ Beta	giskard-checks	Testing & evaluation — scenario API, built-in checks, LLM-as-judge
✅ Beta	giskard-scan	Agent vulnerability scanner — red teaming, prompt injection, data leakage (successor of v2 Scan)
📋 Planned	giskard-rag	RAG evaluation & synthetic data generation (successor of v2 RAGET)

Giskard Checks — create and apply evals for testing agents

pip install giskard-checks

Giskard Checks is a lightweight library for creating evaluations (evals) that test LLM-based systems — from simple assertions to LLM-as-judge assessments. Unlike traditional unit tests, evals are designed for non-deterministic outputs where the same input can produce different valid responses.

Use Giskard Checks to:

• Catch regressions — verify your system still behaves correctly after changes
• Validate RAG quality — check if answers are grounded in retrieved context
• Enforce safety rules — ensure outputs conform to your content policies
• Evaluate multi-turn agents — test full conversations, not just single exchanges

Built-in evals include string matching, comparisons, regex, semantic similarity, and LLM-as-judge checks (Groundedness, Conformity, LLMJudge).

Quickstart

f