mteb

embeddings-benchmark/mteb

State-of-the-art evaluation of embeddings across languages and modalities

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

46views this month

3.4k stars704 forksLast push Sep 19, 2026 Python Apache-2.0

Decision brief

MTEB is an evaluator for embedding models across languages and modalities under the Apache-2.0 license.

Good fit when

You require benchmarking tools specifically designed for state-of-the-art embedding evaluations in low-resource NLP contexts.
Evaluations spanning multiple tasks like semantic search, clustering, and information retrieval are needed.

Avoid when

Your project exclusively focuses on a single language or modality not covered by MTEB’s broad scope.
You need a tool that supports operations beyond evaluation, such as model training or fine-tuning directly within the same system.

Observed Jul 12, 2026 · Source: enrich:decision_facts

mteb alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Very active (0d since push); As of Sep 20, 2026
Provenance: Not a fork · Organization account; As of Sep 20, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install mteb

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

deepeval

LLM Evaluation Framework.

18k★

Awesome-Multimodal-Large-Language-Models

Latest Advances on Multimodal Large Language Models

18k★

lm-evaluation-harness

A framework for few-shot evaluation of language models.

14k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

MTEB is designed for evaluating the performance of various embedding models across different tasks and languages. It supports a broad spectrum of applications including semantic search, clustering, and information retrieval.

Capability facts

Deploy: Self-host
Source: dockerfile:Dockerfile · Sep 20, 2026
Docker: Dockerfile present
Source: dockerfile:Dockerfile · Sep 20, 2026
CLI: CLI entrypoint
Source: pyproject.toml:[project.scripts] · Sep 20, 2026
Languages: python
Source: github.language+pyproject.toml · Sep 20, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Sep 20, 2026)

pip install mteb

Source link

README

Installation You can install mteb simply using pip or uv. For more on installation please see the documentation. For faster installation, you can also use uv:

For agents

This page has a .md twin and JSON over the API.

/tools/embeddings-benchmark-mteb.md GET /api/graphcanon/tools/embeddings-benchmark-mteb GET /api/graphcanon/graph?tool=embeddings-benchmark-mteb