VLMEvalKit
open-compass/VLMEvalKit
An open-source evaluation toolkit for large vision-language models (LVLMs) supporting over 220 LMMs and 80+ benchmarks.
Overview
VLMEvalKit enables one-command evaluation of LVLMs on various benchmarks, adopting generation-based evaluation for all LVLMs and providing results with exact matching and LLM-based answer extraction. It supports models in thinking mode through a custom `split_thinking` function.
Categories
Tags
Similar tools
ollama
ollama/ollama
Local inference runtime and CLI for open-weight large language models
transformers
huggingface/transformers
huggingface/transformers
JavaGuide
Snailclimb/JavaGuide
Java guide for backend interviews & AI application development covering system design, LLMs, Agents, and RAG.
langflow
langflow-ai/langflow
Metadata derived from provided repository information.
langchain
langchain-ai/langchain
The agent engineering platform.
LLMs-from-scratch
rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch
Install
pip install VLMEvalKitREADME
A Toolkit for Evaluating Large Vision-Language Models.
[![][github-contributors-shield]][github-contributors-link] • [![][github-forks-shield]][github-forks-link] • [![][github-stars-shield]][github-stars-link] • [![][github-issues-shield]][github-issues-link] • [![][github-license-shield]][github-license-link]
English | 简体中文 | 日本語
🏆 OC Learderboard • 🏗️Quickstart • 📊Datasets & Models • 🛠️Development
🤗 HF Leaderboard • 🤗 Evaluation Records • 🤗 HF Video Leaderboard •
🔊 Discord • 📝 Report • 🎯Goal • 🖊️Citation
VLMEvalKit (the python package name is vlmeval) is an open-source evaluation toolkit of large vision-language models (LVLMs). It enables one-command evaluation of LVLMs on various benchmarks, without the heavy workload of data preparation under multiple repositories. In VLMEvalKit, we adopt generation-based evaluation for all LVLMs, and provide the evaluation results obtained with both exact matching and LLM-based answer extraction.
Recent Codebase Changes
-
[2025-09-12] Major Update: Improved Handling for Models with Thinking Mode
A new feature in PR 1229 that improves support for models with thinking mode. VLMEvalKit now allows for the use of a custom
split_thinkingfunction. We strongly recommend this for models with thinking mode to ensure the accuracy of evaluation. To use this new functionality, please enable the Environment Variable:SPLIT_THINK=True. By default, the function will parse content within<think>...</think>tags and store it in thethinkingkey of the output. For more advanced customization, you can also create asplit_thinkfunction for model. Please see the InternVL implementation for an example. -
[2025-09-12] Major Update: Improved Handling for Long Response(More than 16k/32k)
A new feature in PR 1229 that improves support for models with long response outputs. VLMEvalKit can now save prediction files in TSV format. Since individual cells in an
.xlsxfile are limited to 32,767 characters, we strongly recommend using this feature for models that generate long responses (e.g., exceeding 16k or 32k tokens) to prevent data truncation. To use this new functionality, please enable the Environment Variable:PRED_FORMAT=tsv. -
[2025-08-04] In PR 1175, we refine the
can_infer_optionandcan_infer_text, which increasingly route the evaluation to LLM choice extractors and empirically leads to slight performance improvement for MCQ benchmarks.
🆕 News
- [2026-04-08] Supported Video-MME-v2. Video-MME-v2 is an authoritative benchmark towards the next stage in video understanding evaluation. 🔥🔥🔥
- [2025-07-07] Supported SeePhys, which is a full spectrum multimodal benchmark for evaluating physics reasoning across different knowledge levels. thanks to Quinn777 🔥🔥🔥
- [2025-07-02] Supported OvisU1, thanks to liyang-7 🔥🔥🔥
- [2025-06-16] Supported [**