Home/AI Agents/judgeval

judgeval

JudgmentLabs/judgeval

The Continuous-Improvement Stack for Agents

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

20views this month

1.1k stars99 forksLast push Sep 16, 2026 Python Apache-2.0

Decision brief

Judgeval is a Python tool that aids in the continuous improvement of AI agents through comprehensive environment data and evaluations, supporting methodologies like reinforcement learning and prompt engineering.

Good fit when

You are working on an AI project where continuous monitoring and enhancement of your agent's performance are critical.
Your project involves complex environments or tasks requiring sophisticated evaluation frameworks such as reinforcement learning or langchain integration.

Avoid when

If you are looking for a tool focused solely on the theoretical aspects of AI development without practical, continuous improvement methodologies.
You require a solution that only supports evaluation metrics and does not offer integrated environment data support, diverging from Judgeval’s comprehensive approach.

Observed Jul 12, 2026 · Source: enrich:decision_facts

judgeval alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Very active (4d since push); As of Sep 20, 2026
Provenance: Not a fork · Organization account; As of Sep 20, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install judgeval

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

evals

Framework for evaluating LLMs and LLM systems with an open-source registry of benchmarks.

19k★

agentdojo

A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

893★

continuous-eval

Data-Driven Evaluation for LLM-Powered Applications

517★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

JudgmentLabs/judgeval is a repository focused on enhancing and monitoring AI agents through environment data and evaluations. It supports various frameworks and methodologies like reinforcement learning, langchain, and prompt engineering.

Capability facts

CLI: CLI entrypoint
Source: pyproject.toml:[project.scripts] · Sep 20, 2026
Languages: python
Source: github.language+pyproject.toml · Sep 20, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

LangGraph integrationLangGraph

Source: README excerpt (regex_v1, Sep 20, 2026)

ion for OpenAI, Anthropic, Google GenAI, and Together AI. Framework support for LangGraph, OpenLit, and Claude Agent SDK.

Source link

Python runtimePython

Source: README excerpt (regex_v1, Sep 20, 2026)

Judgeval is an open-source Python SDK for agent improvement. It provides tracing and agent-judge evaluation for LL

Source link

README

<div align="center" <a href="https://judgmentlabs.ai/" <picture <source media="(prefers color scheme: dark)" srcset="assets/logo darkmode.svg" <img src="assets/logo lightmode.svg" alt="Judgment Logo" width="400" / </picture </a <br The Continuous Improvement Stack for Agents Dete...

For agents

This page has a .md twin and JSON over the API.

/tools/judgmentlabs-judgeval.md GET /api/graphcanon/tools/judgmentlabs-judgeval GET /api/graphcanon/graph?tool=judgmentlabs-judgeval