{"data":{"slug":"openai-simple-evals","name":"simple-evals","tagline":"simple-evals","github_url":"https://github.com/openai/simple-evals","owner":"openai","repo":"simple-evals","owner_avatar_url":"https://avatars.githubusercontent.com/u/14957082?v=4","primary_language":"Python","stars":4565,"forks":493,"topics":[],"archived":false,"github_pushed_at":"2026-04-22T22:16:18+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/openai-simple-evals","markdown_url":"https://www.graphcanon.com/tools/openai-simple-evals.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/openai-simple-evals","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=openai-simple-evals","description":null,"homepage_url":null,"license":"MIT","open_issues":56,"watchers":49,"ai_summary":null,"readme_excerpt":"# ⚠️ Deprecation Notice\n\n**July 2025**: `simple-evals` will no longer be updated for new models or benchmark results. The repo will continue to host reference implementations for **HealthBench**, **BrowseComp**, and **SimpleQA**.\n\n---\n\n# Overview\nThis repository contains a lightweight library for evaluating language models.\nWe are open sourcing it so we can be transparent about the accuracy numbers we're publishing alongside our latest models.\n\n## Benchmark Results\n\n| Model                        | Prompt        | MMLU   | GPQA [^8]   | MATH [^6]| HumanEval | MGSM[^5] | DROP[^5]<br>(F1, 3-shot) | SimpleQA\n|:----------------------------:|:-------------:|:------:|:------:|:--------:|:---------:|:------:|:--------------------------:|:---------:|\n| **o3**                         |               |        |        |          |           |        |                             |                      |           |\n| o3-high [^10]                | n/a [^7]      |  93.3  |  83.4  |   98.1   |  88.4     |  92.0  |  89.8                      |  48.6     |\n| o3 [^9] [^10]                | n/a           |  92.9  |  82.8  |   97.8   |  87.4     |  92.3  |  80.6                      |  49.4     |\n| o3-low [^10]                 | n/a           |  92.8  |  78.6  |   96.9   |  87.3     |  91.9  |  82.3                      |  49.4     |\n| **o4-mini**                    |               |        |        |          |           |        |                             |                      |\n| o4-mini-high [^9] [^10]      | n/a           |  90.3  |  81.3  |   98.2   |  99.3     |  93.5  |  78.1                      |  19.3     |\n| o4-mini [^9] [^10]           | n/a           |  90.0  |  77.6  |   97.5   |  97.3     |  93.7  |  77.7                      |  20.2     |\n| o4-mini-low [^10]            | n/a           |  89.5  |  73.6  |   96.2   |  95.9     |  93.0  |  76.0                      |  20.2     |\n| **o3-mini**                    |               |        |        |          |           |        |                             |                      |           |\n| o3-mini-high                 | n/a           |  86.9  |  77.2  |   97.9   |  97.6     |  92.0  |  80.6                      |  13.8     |\n| o3-mini                      | n/a           |  85.9  |  74.9  |   97.3   |  96.3     |  90.8  |  79.2                      |  13.4     |\n| o3-mini-low                  | n/a           |  84.9  |  67.6  |   95.8   |  94.5     |  89.4  |  77.6                      |  13.0     |\n| **o1**                         |               |        |        |          |           |        |                             |                      |\n|  o1                          | n/a           |  91.8  |  75.7  |   96.4   |    -      |  89.3  |  90.2                      |  42.6     |\n| o1-preview                   | n/a           |  90.8  |  73.3  |   85.5   |  92.4     |  90.8  |  74.8                      |  42.4     |\n| o1-mini                      | n/a           |  85.2  |  60.0  |   90.0   |  92.4     |  89.9  |  83.9                      |  07.6     |\n| **GPT-4.1**                            |               |        |        |          |           |        |                             |                      |           |\n| gpt-4.1-2025-04-14           | assistant [^2]|  90.2  |  66.3  |   82.1   |   94.5    |  86.9  |  79.4                      | 41.6      |\n| gpt-4.1-mini-2025-04-14      | assistant     |  87.5  |  65.0  |   81.4   |   93.8    |  88.2  |  81.0                      | 16.8      |\n| gpt-4.1-nano-2025-04-14      | assistant     |  80.1  |  50.3  |   62.3   |   87.0    |  73.0  |  82.2                      | 07.6      |\n| **GPT-4o**                     |               |        |        |          |           |        |                             |                      |           |\n| gpt-4o-2024-11-20            | assistant     |  85.7  |  46.0  |   68.5   |   90.2    |  90.3  |  81.5                      | 38.8      |\n| gpt-4o-2024-08-06","github_created_at":"2024-04-11T22:38:17+00:00","created_at":"2026-07-11T10:34:42.006436+00:00","updated_at":"2026-07-11T20:37:12.57167+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":785446833,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:34:42.628Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":0,"days_since_push":79,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:48.067Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:34:47.706Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T10:34:47.706Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T10:34:47.706Z"}}}}