{"data":{"slug":"openai-simple-evals","name":"simple-evals","tagline":"A lightweight library for evaluating language models.","github_url":"https://github.com/openai/simple-evals","owner":"openai","repo":"simple-evals","owner_avatar_url":"https://avatars.githubusercontent.com/u/14957082?v=4","primary_language":"Python","stars":4658,"forks":513,"topics":[],"archived":false,"github_pushed_at":"2026-04-22T22:16:18+00:00","maintenance_label":"Slowing","stars_delta_30d":33,"url":"https://www.graphcanon.com/tools/openai-simple-evals","markdown_url":"https://www.graphcanon.com/tools/openai-simple-evals.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/openai-simple-evals","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=openai-simple-evals","description":null,"homepage_url":null,"license":"MIT","open_issues":66,"watchers":47,"ai_summary":"This repository hosts reference implementations for benchmarking language models including HealthBench, BrowseComp, and SimpleQA. It provides tools to evaluate model performance across various tasks but will not receive updates for new models or benchmark results beyond July 2025.","readme_excerpt":"# ⚠️ Deprecation Notice\n\n**July 2025**: `simple-evals` will no longer be updated for new models or benchmark results. The repo will continue to host reference implementations for **HealthBench**, **BrowseComp**, and **SimpleQA**.\n\n---\n\n# Overview\nThis repository contains a lightweight library for evaluating language models.\nWe are open sourcing it so we can be transparent about the accuracy numbers we're publishing alongside our latest models.\n\n## Benchmark Results\n\n| Model | Prompt | MMLU | GPQA [^8] | MATH [^6]| HumanEval | MGSM[^5] | DROP[^5]
(F1, 3-shot) | SimpleQA\n|:----------------------------:|:-------------:|:------:|:------:|:--------:|:---------:|:------:|:--------------------------:|:---------:|\n| **o3** | | | | | | | | | |\n| o3-high [^10] | n/a [^7] | 93.3 | 83.4 | 98.1 | 88.4 | 92.0 | 89.8 | 48.6 |\n| o3 [^9] [^10] | n/a | 92.9 | 82.8 | 97.8 | 87.4 | 92.3 | 80.6 | 49.4 |\n| o3-low [^10] | n/a | 92.8 | 78.6 | 96.9 | 87.3 | 91.9 | 82.3 | 49.4 |\n| **o4-mini** | | | | | | | | |\n| o4-mini-high [^9] [^10] | n/a | 90.3 | 81.3 | 98.2 | 99.3 | 93.5 | 78.1 | 19.3 |\n| o4-mini [^9] [^10] | n/a | 90.0 | 77.6 | 97.5 | 97.3 | 93.7 | 77.7 | 20.2 |\n| o4-mini-low [^10] | n/a | 89.5 | 73.6 | 96.2 | 95.9 | 93.0 | 76.0 | 20.2 |\n| **o3-mini** | | | | | | | | | |\n| o3-mini-high | n/a | 86.9 | 77.2 | 97.9 | 97.6 | 92.0 | 80.6 | 13.8 |\n| o3-mini | n/a | 85.9 | 74.9 | 97.3 | 96.3 | 90.8 | 79.2 | 13.4 |\n| o3-mini-low | n/a | 84.9 | 67.6 | 95.8 | 94.5 | 89.4 | 77.6 | 13.0 |\n| **o1** | | | | | | | | |\n| o1 | n/a | 91.8 | 75.7 | 96.4 | - | 89.3 | 90.2 | 42.6 |\n| o1-preview | n/a | 90.8 | 73.3 | 85.5 | 92.4 | 90.8 | 74.8 | 42.4 |\n| o1-mini | n/a | 85.2 | 60.0 | 90.0 | 92.4 | 89.9 | 83.9 | 07.6 |\n| **GPT-4.1** | | | | | | | | | |\n| gpt-4.1-2025-04-14 | assistant [^2]| 90.2 | 66.3 | 82.1 | 94.5 | 86.9 | 79.4 | 41.6 |\n| gpt-4.1-mini-2025-04-14 | assistant | 87.5 | 65.0 | 81.4 | 93.8 | 88.2 | 81.0 | 16.8 |\n| gpt-4.1-nano-2025-04-14 | assistant | 80.1 | 50.3 | 62.3 | 87.0 | 73.0 | 82.2 | 07.6 |\n| **GPT-4o** | | | | | | | | | |\n| gpt-4o-2024-11-20 | assistant | 85.7 | 46.0 | 68.5 | 90.2 | 90.3 | 81.5 | 38.8 |\n| gpt-4o-2024-08-06","github_created_at":"2024-04-11T22:38:17+00:00","created_at":"2026-07-11T10:34:42.006436+00:00","updated_at":"2026-10-12T06:00:53.477896+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"benchmark","name":"benchmark"},{"slug":"depreciation-notice","name":"depreciation notice"},{"slug":"evaluation","name":"evaluation"},{"slug":"language-models","name":"language models"}],"trust":{"provenance":{"is_fork":false,"github_id":785446833,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-10-12T06:00:52.645Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":172,"last_release_at":null,"stars_delta_30d":33,"open_issues_delta_30d":4},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:48.067Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-10-12T06:00:53.138Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-10-12T06:00:53.138Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-10-12T06:00:53.138Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need a stable baseline to evaluate model performance with specific benchmarks like MMLU, HumanEval, and DROP that won't change after July 2025","If your project relies on the exact benchmark results for models evaluated in the repository from June 2024 to July 2025"],"when_not_to_use":["For evaluating models released or significantly updated after July 2025, as this tool does not include future benchmarks","When you need a tool that will adapt and expand its benchmark set with emerging model releases and evaluation tasks beyond 2025"],"source":"enrich:decision_facts","observed_at":"2026-07-14T21:17:04.045Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"simple-evals provides lightweight tools for evaluating language models using reference implementations from HealthBench, BrowseComp, SimpleQA. Last updates July 2025."},{"label":"License detail","value":"MIT licensed Python library for transparent language model evaluations with specific benchmark support until July 2025."}]}}