{"data":{"slug":"openai-evals","name":"evals","tagline":"Framework for evaluating LLMs and LLM systems with an open-source registry of benchmarks.","github_url":"https://github.com/openai/evals","owner":"openai","repo":"evals","owner_avatar_url":"https://avatars.githubusercontent.com/u/14957082?v=4","primary_language":"Python","stars":18890,"forks":3017,"topics":[],"archived":false,"github_pushed_at":"2026-04-14T15:29:57+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/openai-evals","markdown_url":"https://www.graphcanon.com/tools/openai-evals.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/openai-evals","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=openai-evals","description":"Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.","homepage_url":null,"license":"Other","open_issues":217,"watchers":281,"ai_summary":"Evals is a framework from OpenAI designed for the evaluation of large language models (LLMs) and systems built using them. It includes a registry of pre-existing evals to test various dimensions of model performance as well as tools to create custom evaluations tailored to specific use cases.","readme_excerpt":"# OpenAI Evals\n\n> You can now configure and run Evals directly in the OpenAI Dashboard. [Get started →](https://platform.openai.com/docs/guides/evals)\n\nEvals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals for use cases you care about. You can also use your data to build private evals which represent the common LLMs patterns in your workflow without exposing any of that data publicly.\n\nIf you are building with LLMs, creating high quality evals is one of the most impactful things you can do. Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case. In the words of [OpenAI's President Greg Brockman](https://twitter.com/gdb/status/1733553161884127435):\n\n<img width=\"596\" alt=\"https://x.com/gdb/status/1733553161884127435?s=20\" src=\"https://github.com/openai/evals/assets/35577566/ce7840ff-43a8-4d88-bb2f-6b207410333b\">\n\n## Setup\n\nTo run evals, you will need to set up and specify your [OpenAI API key](https://platform.openai.com/account/api-keys). After you obtain an API key, specify it using the [`OPENAI_API_KEY` environment variable](https://platform.openai.com/docs/quickstart/step-2-setup-your-api-key). Please be aware of the [costs](https://openai.com/pricing) associated with using the API when running evals. You can also run and create evals using [Weights & Biases](https://wandb.ai/wandb_fc/openai-evals/reports/OpenAI-Evals-Demo-Using-W-B-Prompts-to-Run-Evaluations--Vmlldzo0MTI4ODA3).\n\n**Minimum Required Version: Python 3.9**\n\n### Downloading evals\n\nOur evals registry is stored using [Git-LFS](https://git-lfs.com/). Once you have downloaded and installed LFS, you can fetch the evals (from within your local copy of the evals repo) with:\n```sh\ncd evals\ngit lfs fetch --all\ngit lfs pull\n```\n\nThis will populate all the pointer files under `evals/registry/data`.\n\nYou may just want to fetch data for a select eval. You can achieve this via:\n```sh\ngit lfs fetch --include=evals/registry/data/${your eval}\ngit lfs pull\n```\n\n### Making evals\n\nIf you are going to be creating evals, we suggest cloning this repo directly from GitHub and installing the requirements using the following command:\n\n```sh\npip install -e .\n```\n\nUsing `-e`, changes you make to your eval will be reflected immediately without having to reinstall.\n\nOptionally, you can install the formatters for pre-committing with:\n\n```sh\npip install -e .[formatters]\n```\n\nThen run `pre-commit install` to install pre-commit into your git hooks. pre-commit will now run on every commit.\n\nIf you want to manually run all pre-commit hooks on a repository, run `pre-commit run --all-files`. To run individual hooks use `pre-commit run <hook_id>`.\n\n## Running evals\n\nIf you don't want to contribute new evals, but simply want to run them locally, you can install the evals package via pip:\n\n```sh\npip install evals\n```\n\nYou can find the full instructions to run existing evals in [`run-evals.md`](docs/run-evals.md) and our existing eval templates in [`eval-templates.md`](docs/eval-templates.md). For more advanced use cases like prompt chains or tool-using agents, you can use our [Completion Function Protocol](docs/completion-fns.md).\n\nWe provide the option for you to log your eval results to a Snowflake database, if you have one or wish to set one up. For this option, you will further have to specify the `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_DATABASE`, `SNOWFLAKE_USERNAME`, and `SNOWFLAKE_PASSWORD` environment variables.\n\n## Writing evals\n\nWe suggest getting starting by: \n\n- Walking through the process for building an eval: [`build-eval.md`](docs/build-eval.md)\n- Exploring an example of implementing custom eval logic: [`custom-eval.md`](docs/custom-eval.md)\n- Writing your own completion functions: [`completion-fns.md`](docs/completion-fns.md)\n- Review our starter g","github_created_at":"2023-01-23T20:51:04+00:00","created_at":"2026-07-11T10:41:03.556915+00:00","updated_at":"2026-07-11T14:25:51.25854+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"benchmarking","name":"benchmarking"},{"slug":"custom-eval-creation","name":"custom eval creation"},{"slug":"evaluation-framework","name":"evaluation framework"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"llm-systems","name":"llm systems"},{"slug":"open-source","name":"open-source"},{"slug":"use-case-testing","name":"use case testing"}],"trust":{"provenance":{"is_fork":false,"github_id":592489166,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:41:04.129Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":0,"days_since_push":87,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:41:05.077Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T14:25:21.629Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-11T14:25:21.629Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T14:25:21.629Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-07-11T14:25:21.629Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["* When you need a comprehensive set of pre-existing evals and the ability to create your own tailored tests using specific use cases, especially within the OpenAI model ecosystem.","* If you are working extensively with LLMs and want to understand how different model versions impact your applications before potentially incurring higher costs with real-world deployments."],"when_not_to_use":["* When evaluating models or systems that do not benefit from being integrated with the OpenAI API, as some features like direct evals configuration in the OpenAI Dashboard require an OpenAI key.","* If you are looking for an evaluation framework that doesn’t involve external dependencies such as Git Large File Storage (LFS) and specific Python version requirements (Python 3.9 minimum), or if a繁"],"source":"enrich:decision_facts","observed_at":"2026-07-11T14:25:50.931Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Evals is an evaluation framework from OpenAI for assessing large language models and systems built with them. It includes an open-source registry of benchmarks and tools to create custom evaluations."}]}}