evals
Framework for evaluating LLMs and LLM systems with an open-source registry of benchmarks.
GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026
67views this month
Decision brief
Evals is an evaluation framework from OpenAI for assessing large language models and systems built with them. It includes an open-source registry of benchmarks and tools to create custom evaluations.
Good fit when
- * When you need a comprehensive set of pre-existing evals and the ability to create your own tailored tests using specific use cases, especially within the OpenAI model ecosystem.
- * If you are working extensively with LLMs and want to understand how different model versions impact your applications before potentially incurring higher costs with real-world deployments.
Avoid when
- * When evaluating models or systems that do not benefit from being integrated with the OpenAI API, as some features like direct evals configuration in the OpenAI Dashboard require an OpenAI key.
- * If you are looking for an evaluation framework that doesn’t involve external dependencies such as Git Large File Storage (LFS) and specific Python version requirements (Python 3.9 minimum), or if a繁
Observed Jul 11, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (145d since push)
- As of Sep 7, 2026
- Provenance
- Not a fork · Organization account
- As of Sep 7, 2026
- Security (OSV)
- No lockfile
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for OpenAI. Display-only - separate from trust and ranking.
- Company
- OpenAI·GitHub org profile·Jul 11, 2026
- Employees
- 4,500·Wikidata (P1128 employees)·Jul 11, 2026
- Funding
- $13,000,000,000 (2023-01)·GraphCanon curated seed (public press)·Jul 11, 2026
- Commercial model
- OSS + managed cloud·GraphCanon curated seed·Jul 11, 2026
Install
pip install evalsSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Evals is a framework from OpenAI designed for the evaluation of large language models (LLMs) and systems built using them. It includes a registry of pre-existing evals to test various dimensions of model performance as well as tools to create custom evaluations tailored to specific use cases.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Sep 7, 2026
- Languages
- python
Source: github.language+pyproject.toml · Sep 7, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Sep 7, 2026)
To run evals, you will need to set up and specify your [OpenAI API key](https://platform.openai.com/account/api-keys). After you obtain an API keySource link
Source: README excerpt (regex_v1, Sep 7, 2026)
**Minimum Required Version: Python 3.9**Source link
Tags
README
OpenAI Evals You can now configure and run Evals directly in the OpenAI Dashboard. Get started → Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. We offer an existing registry of evals to test different dimensions of OpenAI models...
For agents
This page has a .md twin and JSON over the API.