---
title: "lmms-eval"
type: "tool"
slug: "evolvinglmms-lab-lmms-eval"
canonical_url: "https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval"
github_url: "https://github.com/EvolvingLMMs-Lab/lmms-eval"
homepage_url: "https://www.lmms-lab.com"
stars: 4290
forks: 613
primary_language: "Python"
license: "Other"
categories: ["evaluation-observability"]
tags: ["evaluation", "benchmark", "large-language-models", "multimodal-evaluation", "audio-evaluation", "agi", "multimodal", "llm-evaluation"]
updated_at: "2026-07-07T18:41:20.000507+00:00"
---

# lmms-eval

> Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

lmms-eval is an evaluation toolkit designed to assess large language models (LLMs) across text, image, video, and audio tasks. It addresses fragmentation in the multimodal evaluation ecosystem by providing unified benchmarks.

## Facts

- Repository: https://github.com/EvolvingLMMs-Lab/lmms-eval
- Homepage: https://www.lmms-lab.com
- Stars: 4,290 · Forks: 613 · Open issues: 43 · Watchers: 8
- Primary language: Python
- License: Other
- Last pushed: 2026-07-07T01:45:14+00:00

## Categories

- [Evaluation & Observability](/categories/evaluation-observability.md)

## Tags

evaluation, benchmark, large language models, multimodal-evaluation, audio-evaluation, agi, multimodal, llm-evaluation

## Related tools

- [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful platform for building and deploying AI-powered agents and workflows. (★ 151,298)
- [graphrag](/tools/microsoft-graphrag.md) - A modular graph-based Retrieval-Augmented Generation (RAG) system (★ 34,239)
- [langfuse](/tools/langfuse-langfuse.md) - Langfuse: Open source AI engineering platform for LLM evaluation, observability, and prompt management. (★ 30,626)
- [RAG_Techniques](/tools/nirdiamant-rag-techniques.md) - Advanced RAG Techniques (★ 28,398)
- [mlflow](/tools/mlflow-mlflow.md) - The open source AI engineering platform for agents, LLMs, and ML models. (★ 26,921)
- [llm-action](/tools/liguodongiot-llm-action.md) - 分享大模型技术原理和实战经验，涵盖工程化、应用落地 (★ 24,670)
- [awesome-n8n-templates](/tools/enescingoz-awesome-n8n-templates.md) - The largest open-source collection of n8n automation templates on GitHub. (★ 23,714)
- [promptfoo](/tools/promptfoo-promptfoo.md) - Test your prompts, agents, and RAGs. Evaluate LLM performance with red teaming/pentesting/vulnerability scanning. (★ 23,003)

## README (excerpt)

```text
<p align="center" width="70%">
<img src="https://i.postimg.cc/KvkLzbF9/WX20241212-014400-2x.png">
</p>

# LMMs-Eval: Probing Intelligence in the Real World







> We are building the unified evaluation toolkit for frontier models and probing the abilities in real world, shape what we build next.

<details>
<summary>🌐 Available in 17 languages</summary>

[简体中文](docs/i18n/README_zh-CN.md) | [繁體中文](docs/i18n/README_zh-TW.md) | [日本語](docs/i18n/README_ja.md) | [한국어](docs/i18n/README_ko.md) | [Español](docs/i18n/README_es.md) | [Français](docs/i18n/README_fr.md) | [Deutsch](docs/i18n/README_de.md) | [Português](docs/i18n/README_pt-BR.md) | [Русский](docs/i18n/README_ru.md) | [Italiano](docs/i18n/README_it.md) | [Nederlands](docs/i18n/README_nl.md) | [Polski](docs/i18n/README_pl.md) | [Türkçe](docs/i18n/README_tr.md) | [العربية](docs/i18n/README_ar.md) | [हिन्दी](docs/i18n/README_hi.md) | [Tiếng Việt](docs/i18n/README_vi.md) | [Indonesia](docs/i18n/README_id.md)

</details>

📚 [Documentation](docs/README.md) | 📖 [100+ Tasks](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/advanced/current_tasks.md) | 🌟 [30+ Models](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/models) | ⚡ [Quickstart](docs/getting-started/quickstart.md)

🏠 [Homepage](https://www.lmms-lab.com/) | 💬 [Discord](https://discord.gg/8xTM6jWnXa) | 🤝 [Contributing](CONTRIBUTING.md)

---

## Why `lmms-eval`?

Benchmarks decide what gets built next. A model team that trusts its eval numbers can focus on real improvements instead of chasing noise. But the multimodal evaluation ecosystem is fragmented - scattered datasets, inconsistent post-processing, and single-number accuracy scores that hide whether a gain is real or random. Two teams evaluating the same model on the same benchmark routinely report different results.

We believe [better evals lead to better models](https://arxiv.org/pdf/2211.09110). Good evaluation maps the border of what models can do and shapes what we build next.

We are building `lmms-eval` and focusing on three core principles:

- **Reproducible** - One pipeline, deterministic results. Same model, same benchmark, same numbers, every time.
- **Efficient** - Evaluation should not be the bottleneck, even at large scale. Async serving, adaptive batching, and video I/O optimizations keep your GPUs saturated end to end.
- **Trustworthy** - Not just accuracy. Confidence intervals, clustered standard errors, paired comparisons, and ongoing research into evaluation methodology. Results you can trust enough to act on.

For how the pipeline works and the concrete mechanisms behind these principles, see [How the Evaluation Pipeline Works](docs/README.md#how-the-evaluation-pipeline-works) and [Why it's Efficient and Trustworthy](docs/README.md#why-its-efficient-and-trustworthy).

## What's New

**v0.7** (Feb 2026) - Operational simplicity and pipeline maturity. 25+ new tasks across 8 domains, 2 new model backends, agentic task evaluation (`generate_until_agentic`), video I/O overhaul with TorchCodec (up to 3.58x faster), Lance-backed video distribution on Hugging Face, safety/red-teaming baselines, efficiency metrics (per-sample token counts, run-level throughput), and streamlined flattened JSONL log output for cleaner post-analysis. [Release notes](docs/releases/lmms-eval-0.7.md) | [Changelog](docs/releases/CHANGELOG.md).

**v0.6** (Feb 2026) - Evaluation as a service. Standalone HTTP eval server, ~7.5x throughput over v0.5, statistically grounded results (CI, paired t-test), 50+ new tasks. [Release notes](docs/releases/lmms-eval-0.6.md) | [Changelog](docs/releases/CHANGELOG.md).

**v0.5** (Oct 2025) - Audio expansion. Comprehensive audio evaluation, response caching, 50+ benchmark variants across audio, vision, and reasoning. [Release notes](docs/releases/lmms-eval-0.5.md).

<details>
<summary>Older updates</summary>

- [2025-01] [Video-MMMU](https://arxiv.org/abs/2501.13826) - Knowledge acquisition from multi-discipline pr
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/evolvinglmms-lab-lmms-eval`](/api/graphcanon/tools/evolvinglmms-lab-lmms-eval)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_