GraphCanon
Home/Evaluation & Observability/VLMEvalKit
VLMEvalKit logo

VLMEvalKit

open-compass/VLMEvalKit

An open-source evaluation toolkit for large vision-language models

GitHub Website

GraphCanon updated 3d · GitHub synced 3d · 29 views this month

4.3k stars745 forksLast push 3d Python Apache-2.0

Decision brief

VLMEvalKit is an open-source Python evaluation toolkit for large vision-language models that offers one-command evaluation with support for various benchmarks and models.

Good fit when

  • When you need to evaluate models supporting thinking mode, as it provides a custom split_thinking function improving accuracy.
  • For handling long response outputs exceeding 16k or 32k tokens without data truncation by enabling TSV format saving for prediction files.

Avoid when

  • If your project requires evaluation tools that generate Excel files with individual cells larger than the default support of 32,767 characters and cannot switch to TSV format.
  • When you do not need generation-based evaluation methods with exact matching and LLM-based answer extraction.

Observed Jul 14, 2026 · Source: enrich:decision_facts

VLMEvalKit alternativesTrust report
VLMEvalKit vs lmms-eval

Verify the decision

Maintenance and security

Full trust report
Maintenance
Very active (0d since push)
As of 3d
Provenance
Not a fork · Organization account
As of 3d
Security (OSV)
16 low (16 low)
As of 1mo

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install VLMEvalKit
PyPI

How it fits your stack(9)

Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.

Relationship graph

Optional deeper exploration of typed edges and category neighbours.

Similar tools

Same-category neighbours not already linked as typed edges.

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

VLMEvalKit is an open-source Python-based evaluation toolkit for large vision-language models (LVLMs). It allows for streamlined evaluation on various benchmarks without the heavy workload of data preparation.

Capability facts

Languages
python

Source: github.language · Aug 17, 2026

Categories

Evaluation & Observability

Graph entities

Smolagentsframework

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Aug 17, 2026)

**VLMEvalKit** (the python package name is **vlmeval**) is an **open-source evaluation toolkit** of **large
Source link

Tags

computer-visionevaluationlarge language modelsllmmulti-modal

README

A Toolkit for Evaluating Large Vision-Language Models.

[![][github-contributors-shield]][github-contributors-link] • [![][github-forks-shield]][github-forks-link] • [![][github-stars-shield]][github-stars-link] • [![][github-issues-shield]][github-issues-link] • [![][github-license-shield]][github-license-link]

English | 简体中文 | 日本語

🏆 OC Learderboard • 🏗️Quickstart • 📊Datasets & Models • 🛠️Development

🤗 HF Leaderboard • 🤗 Evaluation Records • 🤗 HF Video Leaderboard •

🔊 Discord • 📝 Report • 🎯Goal • 🖊️Citation

VLMEvalKit (the python package name is vlmeval) is an open-source evaluation toolkit of large vision-language models (LVLMs). It enables one-command evaluation of LVLMs on various benchmarks, without the heavy workload of data preparation under multiple repositories. In VLMEvalKit, we adopt generation-based evaluation for all LVLMs, and provide the evaluation results obtained with both exact matching and LLM-based answer extraction.

Recent Codebase Changes

  • [2025-09-12] Major Update: Improved Handling for Models with Thinking Mode

    A new feature in PR 1229 that improves support for models with thinking mode. VLMEvalKit now allows for the use of a custom split_thinking function. We strongly recommend this for models with thinking mode to ensure the accuracy of evaluation. To use this new functionality, please enable the Environment Variable: SPLIT_THINK=True. By default, the function will parse content within <think>...</think> tags and store it in the thinking key of the output. For more advanced customization, you can also create a split_think function for model. Please see the InternVL implementation for an example.

  • [2025-09-12] Major Update: Improved Handling for Long Response(More than 16k/32k)

    A new feature in PR 1229 that improves support for models with long response outputs. VLMEvalKit can now save prediction files in TSV format. Since individual cells in an .xlsx file are limited to 32,767 characters, we strongly recommend using this feature for models that generate long responses (e.g., exceeding 16k or 32k tokens) to prevent data truncation. To use this new functionality, please enable the Environment Variable: PRED_FORMAT=tsv.

  • [2025-08-04] In PR 1175, we refine the can_infer_option and can_infer_text, which increasingly route the evaluation to LLM choice extractors and empirically leads to slight performance improvement for MCQ benchmarks.

🆕 News

  • [2026-04-08] Supported Video-MME-v2. Video-MME-v2 is an authoritative benchmark towards the next stage in video understanding evaluation. 🔥🔥🔥
  • [2025-07-07] Supported SeePhys, which is a ​full spectrum multimodal benchmark for evaluating physics reasoning across different knowledge levels. thanks to Quinn777 🔥🔥🔥
  • [2025-07-02] Supported OvisU1, thanks to liyang-7 🔥🔥🔥
  • [2025-06-16] Supported [**

For agents

This page has a .md twin and JSON over the API.

/tools/open-compass-vlmevalkit.mdGET /api/graphcanon/tools/open-compass-vlmevalkitGET /api/graphcanon/graph?tool=open-compass-vlmevalkit

Was this helpful?

Anonymous feedback helps us improve pages and translations.