⚠️ Deprecation Notice

July 2025: simple-evals will no longer be updated for new models or benchmark results. The repo will continue to host reference implementations for HealthBench, BrowseComp, and SimpleQA.

---

Overview

This repository contains a lightweight library for evaluating language models. We are open sourcing it so we can be transparent about the accuracy numbers we're publishing alongside our latest models.

Benchmark Results

Model	Prompt	MMLU	GPQA [^8]	MATH [^6]	HumanEval	MGSM[^5]	DROP[^5] (F1, 3-shot)	SimpleQA
o3
o3-high [^10]	n/a [^7]	93.3	83.4	98.1	88.4	92.0	89.8	48.6
o3 [^9] [^10]	n/a	92.9	82.8	97.8	87.4	92.3	80.6	49.4
o3-low [^10]	n/a	92.8	78.6	96.9	87.3	91.9	82.3	49.4
o4-mini
o4-mini-high [^9] [^10]	n/a	90.3	81.3	98.2	99.3	93.5	78.1	19.3
o4-mini [^9] [^10]	n/a	90.0	77.6	97.5	97.3	93.7	77.7	20.2
o4-mini-low [^10]	n/a	89.5	73.6	96.2	95.9	93.0	76.0	20.2
o3-mini
o3-mini-high	n/a	86.9	77.2	97.9	97.6	92.0	80.6	13.8
o3-mini	n/a	85.9	74.9	97.3	96.3	90.8	79.2	13.4
o3-mini-low	n/a	84.9	67.6	95.8	94.5	89.4	77.6	13.0
o1
o1	n/a	91.8	75.7	96.4	-	89.3	90.2	42.6
o1-preview	n/a	90.8	73.3	85.5	92.4	90.8	74.8	42.4
o1-mini	n/a	85.2	60.0	90.0	92.4	89.9	83.9	07.6
GPT-4.1
gpt-4.1-2025-04-14	assistant [^2]	90.2	66.3	82.1	94.5	86.9	79.4	41.6
gpt-4.1-mini-2025-04-14	assistant	87.5	65.0	81.4	93.8	88.2	81.0	16.8
gpt-4.1-nano-2025-04-14	assistant	80.1	50.3	62.3	87.0	73.0	82.2	07.6
GPT-4o
gpt-4o-2024-11-20	assistant	85.7	46.0	68.5	90.2	90.3	81.5	38.8
gpt-4o-2024-08-06