{"data":{"slug":"judgmentlabs-judgeval","name":"judgeval","tagline":"The Continuous-Improvement Stack for Agents","github_url":"https://github.com/JudgmentLabs/judgeval","owner":"JudgmentLabs","repo":"judgeval","owner_avatar_url":"https://avatars.githubusercontent.com/u/186249420?v=4","primary_language":"Python","stars":1063,"forks":99,"topics":["agent","agentic-ai","agents","grpo","langchain","langgraph","llama-index","llm","llm-evaluation","llm-observability","open-source","openai","prompt-engineering","reinforcement-learning","rl"],"archived":false,"github_pushed_at":"2026-09-16T00:31:51+00:00","maintenance_label":"Very active","stars_delta_30d":16,"url":"https://www.graphcanon.com/tools/judgmentlabs-judgeval","markdown_url":"https://www.graphcanon.com/tools/judgmentlabs-judgeval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/judgmentlabs-judgeval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=judgmentlabs-judgeval","description":"The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.","homepage_url":"https://judgmentlabs.ai/","license":"Apache-2.0","open_issues":38,"watchers":7,"ai_summary":"JudgmentLabs/judgeval is a repository focused on enhancing and monitoring AI agents through environment data and evaluations. It supports various frameworks and methodologies like reinforcement learning, langchain, and prompt engineering.","readme_excerpt":"<div align=\"center\">\n\n<a href=\"https://judgmentlabs.ai/\">\n  <picture>\n    <source media=\"(prefers-color-scheme: dark)\" srcset=\"assets/logo_darkmode.svg\">\n    <img src=\"assets/logo_lightmode.svg\" alt=\"Judgment Logo\" width=\"400\" />\n  </picture>\n</a>\n\n<br>\n\n## The Continuous-Improvement Stack for Agents\n\nDetect failures, triage root causes, and ship fixes backed by production data.\n\n\n\n\n\n\n\n</div>\n\n## Overview\n\nJudgeval is an open-source Python SDK for agent improvement. It provides tracing and agent-judge evaluation for LLM-powered applications — so you can detect failures, understand what went wrong, and validate fixes against real production cases before shipping.\n\nTo get started, dive into the [docs](https://docs.judgmentlabs.ai/documentation).\n\n## Why Judgeval\n\n**OpenTelemetry-based tracing** -- Instrument any function with `@Tracer.observe()`. Automatically captures inputs, outputs, and LLM token usage. Built on OpenTelemetry for full compatibility with existing observability stacks.\n\n**Agent judges** -- Define prompt-based scorers to evaluate agent behaviors at scale. Judges produce structured behaviors — scored, labeled outputs that describe how your agent acted — which accumulate into a searchable record of agent behavior over time. Run judges against live production traffic or replay them on historical traces to validate fixes before shipping.\n\n**Online monitoring** -- Automatically score live production traffic server-side with no latency impact. Detected behaviors surface as structured signals — configure Slack alerts so regressions and recurrences never go unnoticed.\n\n**Broad integrations** -- Auto-instrumentation for OpenAI, Anthropic, Google GenAI, and Together AI. Framework support for LangGraph, OpenLit, and Claude Agent SDK.\n\n## Quickstart\n\nInstall the SDK:\n\n```bash\npip install judgeval\n```\n\nSet your credentials:\n\n```bash\nexport JUDGMENT_API_KEY=...\nexport JUDGMENT_ORG_ID=...\n```\n\nAdd observability to your agent with two lines of setup:\n\n```python\nfrom judgeval import Tracer, wrap\nfrom openai import OpenAI\n\nTracer.init(project_name=\"my-project\")\nclient = wrap(OpenAI())\n\n@Tracer.observe(span_type=\"tool\")\ndef search(query: str) -> str:\n    results = vector_db.search(query)\n    return results\n\n@Tracer.observe(span_type=\"agent\")\ndef run_agent(question: str) -> str:\n    context = search(question)\n    response = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=[{\"role\": \"user\", \"content\": f\"{context}\\n\\n{question}\"}],\n    )\n    return response.choices[0].message.content\n\nrun_agent(\"What is the capital of the United States?\")\n```\n\n### JQL\n\nJQL queries use the same API key, organization, and project configuration as the\nrest of Judgeval. Tenant identifiers are not part of the query payload.\n\n```python\nfrom judgeval import Judgeval\nfrom judgeval.jql import spans\n\nclient = Judgeval(project_name=\"my-project\")\nresult = client.query(spans().rows(), trace_ids=[\"trace-123\"])\n```\n\n`trace_ids` and `session_ids` are mutually exclusive options outside the JQL\nquery object. Trace IDs narrow the query directly. Judgment resolves session IDs\nwithin the authenticated organization and project, then narrows every part of\nthe query to their traces. If no session resolves, the request fails instead of\nfalling back to the whole project. Both options work with `present()` and\n`discover()`.\n\nUse `offline_traces()` or `offline_spans()` to query traces captured by an\noffline test:\n\n```python\nfrom judgeval.jql import offline_traces\n\noffline_result = client.query(offline_traces().rows())\n```\n\n## Integrations\n\nSupports OpenAI, Anthropic, Google GenAI, Together AI, LangGraph, OpenLit, and Claude Agent SDK. See the full [integrations docs](https://docs.judgmentlabs.ai/documentation/integrations/introduction).\n\n## CLI\n\nManage agents, traces, judges, behaviors, and evaluations from the terminal. Query trace history, deploy judges, inspect detected behaviors, and run evals against production data — all without leaving your s","github_created_at":"2024-10-25T04:49:15+00:00","created_at":"2026-07-11T11:59:42.803001+00:00","updated_at":"2026-09-20T06:57:25.909506+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"agent","name":"agent"},{"slug":"agentic-ai","name":"agentic-ai"},{"slug":"agents","name":"agents"},{"slug":"grpo","name":"grpo"},{"slug":"langchain","name":"langchain"},{"slug":"llamaindex","name":"llamaindex"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"open-source","name":"open-source"}],"trust":{"provenance":{"is_fork":false,"github_id":878272507,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T06:57:23.559Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":6,"days_since_push":4,"last_release_at":"2026-09-01T21:39:49Z","stars_delta_30d":16,"open_issues_delta_30d":10},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:59:44.036Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T06:57:24.587Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-20T06:57:24.587Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-20T06:57:24.587Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-20T06:57:24.587Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["You are working on an AI project where continuous monitoring and enhancement of your agent's performance are critical.","Your project involves complex environments or tasks requiring sophisticated evaluation frameworks such as reinforcement learning or langchain integration."],"when_not_to_use":["If you are looking for a tool focused solely on the theoretical aspects of AI development without practical, continuous improvement methodologies.","You require a solution that only supports evaluation metrics and does not offer integrated environment data support, diverging from Judgeval’s comprehensive approach."],"source":"enrich:decision_facts","observed_at":"2026-07-12T08:51:45.715Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Judgeval is a Python tool that aids in the continuous improvement of AI agents through comprehensive environment data and evaluations, supporting methodologies like reinforcement learning and prompt engineering."}]}}