{"data":{"slug":"evolvinglmms-lab-lmms-eval","name":"lmms-eval","tagline":"One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks","github_url":"https://github.com/EvolvingLMMs-Lab/lmms-eval","owner":"EvolvingLMMs-Lab","repo":"lmms-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/154951679?v=4","primary_language":"Python","stars":4368,"forks":639,"topics":["agi","audio-evaluation","benchmark","evaluation","large-language-models","llm-evaluation","multimodal","multimodal-evaluation","video-understanding","vision-language-model","vlm"],"archived":false,"github_pushed_at":"2026-08-06T02:22:23+00:00","maintenance_label":"Active","stars_delta_30d":52,"url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval","markdown_url":"https://www.graphcanon.com/tools/evolvinglmms-lab-lmms-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/evolvinglmms-lab-lmms-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=evolvinglmms-lab-lmms-eval","description":"One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks","homepage_url":"https://www.lmms-lab.com","license":"Other","open_issues":49,"watchers":9,"ai_summary":"A comprehensive evaluation toolkit for multimodal large language models including text, image, video, and audio tasks. It supports benchmarking and reproducibility of results across various datasets.","readme_excerpt":"### Alternative Installation\n\nFor direct usage from Git:\n```bash\nuv venv eval\nuv venv --python 3.12\nsource eval/bin/activate\n\n---\n\n# You might need to add and include your own task yaml if using this installation\nuv pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git\n```\n\n
\nReproduction of LLaVA-1.5's paper results\n\nYou can check the [torch environment info](miscs/repr_torch_envs.txt) and [results check](miscs/llava_result_check.md) to **reproduce LLaVA-1.5's paper results**. We found torch/cuda versions difference would cause small variations in the results.\n\n
\n\nIf you want to test on caption dataset such as `coco`, `refcoco`, and `nocaps`, you will need to have `java==1.8.0` to let pycocoeval api to work. If you don't have it, you can install by using conda\n```\nconda install openjdk=8\n```\nyou can then check your java version by `java -version`\n\n\n
\nComprehensive Evaluation Results of LLaVA Family Models\n
\n\nAs demonstrated by the extensive table below, we aim to provide detailed information for readers to understand the datasets included in lmms-eval and some specific details about these datasets (we remain grateful for any corrections readers may have during our evaluation process).\n\nWe provide a Google Sheet for the detailed results of the LLaVA series models on different datasets. You can access the sheet [here](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?usp=sharing). It's a live sheet, and we are updating it with new results.\n\n

\n\n

\n\nWe also provide the raw data exported from Weights & Biases for the detailed results of the LLaVA series models on different datasets. You can access the raw data [here](https://docs.google.com/spreadsheets/d/1AvaEmuG4csSmXaHjgu4ei1KBMmNNW8wflOD_kkTDdv8/edit?usp=sharing).\n\n
\n
\n\nIf you want to test [VILA](https://github.com/NVlabs/VILA), you should install the following dependencies:\n\n```bash\npip install s2wrapper@git+https://github.com/bfshi/scaling_on_scales\n```\n\nOur Development will be continuing on the main branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub.\n\n---\n\n### Requirements\n\n- Node.js 18+ (for building the frontend, auto-built on first run)","github_created_at":"2024-03-07T12:09:25+00:00","created_at":"2026-07-07T17:35:20.843304+00:00","updated_at":"2026-08-17T18:01:04.394379+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"agi","name":"agi"},{"slug":"audio-evaluation","name":"audio-evaluation"},{"slug":"benchmark","name":"benchmark"},{"slug":"evaluation","name":"evaluation"},{"slug":"large-language-models","name":"large language models"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"multimodal","name":"multimodal"},{"slug":"multimodal-evaluation","name":"multimodal-evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":768612800,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-17T18:01:01.828Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":1,"days_since_push":11,"last_release_at":"2026-06-24T08:43:39Z","stars_delta_30d":52,"open_issues_delta_30d":9},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:05:14.783Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-17T18:01:03.096Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-17T18:01:03.096Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-17T18:01:03.096Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-08-17T18:01:03.096Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["You need to evaluate LLaVA series models on different datasets with precise control over reproducibility details like torch/cuda versions.","Require direct access to task YAML files for custom evaluations or adding new modalities.","Need Java 1.8.0 specifically due to pycocoeval dependencies when testing captioning tasks."],"when_not_to_use":["Looking for a tool that supports less than Python 3.12, as uv setup mandates this version.","Requiring support beyond text, image, video, and audio modalities which lmms-eval specifically covers.","Your project doesn't benefit from extensive results tracking in Google Sheets or relies solely on alternative reproducibility mechanisms without external dependencies."],"source":"enrich:decision_facts","observed_at":"2026-07-12T18:26:30.057Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"lmms-eval is a one-stop solution for benchmarking multimodal large language models across various tasks including text, image, video, and audio."}]}}