LMMs-Eval: Probing Intelligence in the Real World

We are building the unified evaluation toolkit for frontier models and probing the abilities in real world, shape what we build next.

🌐 Available in 17 languages

简体中文 | 繁體中文 | 日本語 | 한국어 | Español | Français | Deutsch | Português | Русский | Italiano | Nederlands | Polski | Türkçe | العربية | हिन्दी | Tiếng Việt | Indonesia

📚 Documentation | 📖 100+ Tasks | 🌟 30+ Models | ⚡ Quickstart

🏠 Homepage | 💬 Discord | 🤝 Contributing

---

Why lmms-eval?

Benchmarks decide what gets built next. A model team that trusts its eval numbers can focus on real improvements instead of chasing noise. But the multimodal evaluation ecosystem is fragmented - scattered datasets, inconsistent post-processing, and single-number accuracy scores that hide whether a gain is real or random. Two teams evaluating the same model on the same benchmark routinely report different results.

We believe better evals lead to better models. Good evaluation maps the border of what models can do and shapes what we build next.

We are building lmms-eval and focusing on three core principles:

• Reproducible - One pipeline, deterministic results. Same model, same benchmark, same numbers, every time.
• Efficient - Evaluation should not be the bottleneck, even at large scale. Async serving, adaptive batching, and video I/O optimizations keep your GPUs saturated end to end.
• Trustworthy - Not just accuracy. Confidence intervals, clustered standard errors, paired comparisons, and ongoing research into evaluation methodology. Results you can trust enough to act on.

For how the pipeline works and the concrete mechanisms behind these principles, see How the Evaluation Pipeline Works and Why it's Efficient and Trustworthy.

What's New

v0.7 (Feb 2026) - Operational simplicity and pipeline maturity. 25+ new tasks across 8 domains, 2 new model backends, agentic task evaluation (generate_until_agentic), video I/O overhaul with TorchCodec (up to 3.58x faster), Lance-backed video distribution on Hugging Face, safety/red-teaming baselines, efficiency metrics (per-sample token counts, run-level throughput), and streamlined flattened JSONL log output for cleaner post-analysis. Release notes | Changelog.

v0.6 (Feb 2026) - Evaluation as a service. Standalone HTTP eval server, ~7.5x throughput over v0.5, statistically grounded results (CI, paired t-test), 50+ new tasks. Release notes | Changelog.

v0.5 (Oct 2025) - Audio expansion. Comprehensive audio evaluation, response caching, 50+ benchmark variants across audio, vision, and reasoning. Release notes.

Older updates

• [2025-01] Video-MMMU - Knowledge acquisition from multi-discipline pr