lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
GraphCanon updated 4d · GitHub synced 4d · 28 views this month
Decision brief
lmms-eval is a one-stop solution for benchmarking multimodal large language models across various tasks including text, image, video, and audio.
Good fit when
- You need to evaluate LLaVA series models on different datasets with precise control over reproducibility details like torch/cuda versions.
- Require direct access to task YAML files for custom evaluations or adding new modalities.
Avoid when
- Looking for a tool that supports less than Python 3.12, as uv setup mandates this version.
- Requiring support beyond text, image, video, and audio modalities which lmms-eval specifically covers.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Active (11d since push)
- As of 4d
- Provenance
- Not a fork · Organization account
- As of 4d
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install lmms-evalHow it fits your stack(11)
Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.
Alternative
Integrates
Relationship graph
Optional deeper exploration of typed edges and category neighbours.
Similar tools
Same-category neighbours not already linked as typed edges.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A comprehensive evaluation toolkit for multimodal large language models including text, image, video, and audio tasks. It supports benchmarking and reproducibility of results across various datasets.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 17, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 17, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 17, 2026)
- Node.js 18+ (for building the frontend, auto-built on first run)Source link
Tags
README
Alternative Installation
For direct usage from Git:
uv venv eval
uv venv --python 3.12
source eval/bin/activate
---
# You might need to add and include your own task yaml if using this installation
uv pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
Reproduction of LLaVA-1.5's paper results
You can check the torch environment info and results check to reproduce LLaVA-1.5's paper results. We found torch/cuda versions difference would cause small variations in the results.
If you want to test on caption dataset such as coco, refcoco, and nocaps, you will need to have java==1.8.0 to let pycocoeval api to work. If you don't have it, you can install by using conda
conda install openjdk=8
you can then check your java version by java -version
Comprehensive Evaluation Results of LLaVA Family Models
As demonstrated by the extensive table below, we aim to provide detailed information for readers to understand the datasets included in lmms-eval and some specific details about these datasets (we remain grateful for any corrections readers may have during our evaluation process).
We provide a Google Sheet for the detailed results of the LLaVA series models on different datasets. You can access the sheet here. It's a live sheet, and we are updating it with new results.
We also provide the raw data exported from Weights & Biases for the detailed results of the LLaVA series models on different datasets. You can access the raw data here.
If you want to test VILA, you should install the following dependencies:
pip install s2wrapper@git+https://github.com/bfshi/scaling_on_scales
Our Development will be continuing on the main branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub.
Requirements
- Node.js 18+ (for building the frontend, auto-built on first run)
For agents
This page has a .md twin and JSON over the API.