GraphCanon updated 2w · GitHub synced 2w
Decision brief
simple-evals provides lightweight tools for evaluating language models using reference implementations from HealthBench, BrowseComp, SimpleQA. Last updates July 2025.
Good fit when
- When you need a stable baseline to evaluate model performance with specific benchmarks like MMLU, HumanEval, and DROP that won't change after July 2025
- If your project relies on the exact benchmark results for models evaluated in the repository from June 2024 to July 2025
Avoid when
- For evaluating models released or significantly updated after July 2025, as this tool does not include future benchmarks
- When you need a tool that will adapt and expand its benchmark set with emerging model releases and evaluation tasks beyond 2025
Observed Jul 14, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (106d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for OpenAI. Display-only - separate from trust and ranking.
- Company
- OpenAI·GitHub org profile·1mo
- Employees
- 4,500·Wikidata (P1128 employees)·1mo
- Funding
- $13,000,000,000 (2023-01)·GraphCanon curated seed (public press)·1mo
- Commercial model
- OSS + managed cloud·GraphCanon curated seed·1mo
Install
pip install simple-evalsSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
This repository hosts reference implementations for benchmarking language models including HealthBench, BrowseComp, and SimpleQA. It provides tools to evaluate model performance across various tasks but will not receive updates for new models or benchmark results beyond July 2025.
Capability facts
- Languages
- python
Source: github.language · Aug 7, 2026
Categories
Graph entities
Tags
README
⚠️ Deprecation Notice
July 2025: simple-evals will no longer be updated for new models or benchmark results. The repo will continue to host reference implementations for HealthBench, BrowseComp, and SimpleQA.
Overview
This repository contains a lightweight library for evaluating language models. We are open sourcing it so we can be transparent about the accuracy numbers we're publishing alongside our latest models.
Benchmark Results
| Model | Prompt | MMLU | GPQA [^8] | MATH [^6] | HumanEval | MGSM[^5] | DROP[^5] (F1, 3-shot) | SimpleQA |
|---|---|---|---|---|---|---|---|---|
| o3 | ||||||||
| o3-high [^10] | n/a [^7] | 93.3 | 83.4 | 98.1 | 88.4 | 92.0 | 89.8 | 48.6 |
| o3 [^9] [^10] | n/a | 92.9 | 82.8 | 97.8 | 87.4 | 92.3 | 80.6 | 49.4 |
| o3-low [^10] | n/a | 92.8 | 78.6 | 96.9 | 87.3 | 91.9 | 82.3 | 49.4 |
| o4-mini | ||||||||
| o4-mini-high [^9] [^10] | n/a | 90.3 | 81.3 | 98.2 | 99.3 | 93.5 | 78.1 | 19.3 |
| o4-mini [^9] [^10] | n/a | 90.0 | 77.6 | 97.5 | 97.3 | 93.7 | 77.7 | 20.2 |
| o4-mini-low [^10] | n/a | 89.5 | 73.6 | 96.2 | 95.9 | 93.0 | 76.0 | 20.2 |
| o3-mini | ||||||||
| o3-mini-high | n/a | 86.9 | 77.2 | 97.9 | 97.6 | 92.0 | 80.6 | 13.8 |
| o3-mini | n/a | 85.9 | 74.9 | 97.3 | 96.3 | 90.8 | 79.2 | 13.4 |
| o3-mini-low | n/a | 84.9 | 67.6 | 95.8 | 94.5 | 89.4 | 77.6 | 13.0 |
| o1 | ||||||||
| o1 | n/a | 91.8 | 75.7 | 96.4 | - | 89.3 | 90.2 | 42.6 |
| o1-preview | n/a | 90.8 | 73.3 | 85.5 | 92.4 | 90.8 | 74.8 | 42.4 |
| o1-mini | n/a | 85.2 | 60.0 | 90.0 | 92.4 | 89.9 | 83.9 | 07.6 |
| GPT-4.1 | ||||||||
| gpt-4.1-2025-04-14 | assistant [^2] | 90.2 | 66.3 | 82.1 | 94.5 | 86.9 | 79.4 | 41.6 |
| gpt-4.1-mini-2025-04-14 | assistant | 87.5 | 65.0 | 81.4 | 93.8 | 88.2 | 81.0 | 16.8 |
| gpt-4.1-nano-2025-04-14 | assistant | 80.1 | 50.3 | 62.3 | 87.0 | 73.0 | 82.2 | 07.6 |
| GPT-4o | ||||||||
| gpt-4o-2024-11-20 | assistant | 85.7 | 46.0 | 68.5 | 90.2 | 90.3 | 81.5 | 38.8 |
| gpt-4o-2024-08-06 |
For agents
This page has a .md twin and JSON over the API.