Home/Data & Retrieval/bpemb

bpemb

bheinzerling/bpemb

Pre-trained subword embeddings in 275 languages using Byte-Pair Encoding

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

35views this month

1.2k stars100 forksLast push Oct 1, 2024 Python MIT

Decision brief

bpemb provides pre-trained subword embeddings using Byte-Pair Encoding for up to 275 languages, which can be beneficial in multi-lingual NLP tasks.

Good fit when

When working on multilingual projects that span a vast array of languages (up to 275) where language-specific data is sparse or unavailable
For NLP tasks requiring efficient handling of low-frequency words and rare terms across different languages

Avoid when

If your project focuses solely on high-resource languages like English, Spanish, French where more specialized models provide better performance per task
When the task specifically requires character-level or word-level embeddings and not subword tokenization provided by Byte-Pair Encoding (BPE)

Requirements:: Requires Python environment to operate effectively across various multilingual applications

Observed Jul 12, 2026 · Source: enrich:decision_facts

bpemb alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Dormant (719d since push); As of Sep 20, 2026
Provenance: Not a fork · Personal account; As of Sep 20, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install bpemb

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

lm-evaluation-harness

A framework for few-shot evaluation of language models.

14k★

tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

11k★

bitsandbytes

Large language model quantization toolkit for PyTorch.

8.5k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

A repository containing pre-trained subword embeddings for multiple languages using the Byte-Pair Encoding method, useful for natural language processing tasks.

Capability facts

Languages: python
Source: github.language · Sep 20, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Sep 20, 2026)

```python

Source link

README

BPEmb BPEmb is a collection of pre trained subword embeddings in 275 languages, based on Byte Pair Encoding (BPE) and trained on Wikipedia. Its intended use is as input for neural models in natural language processing. Website ・ Usage ・ Download ・ MultiBPEmb ・ Paper (pdf) ・ Citin...

For agents

This page has a .md twin and JSON over the API.

/tools/bheinzerling-bpemb.md GET /api/graphcanon/tools/bheinzerling-bpemb GET /api/graphcanon/graph?tool=bheinzerling-bpemb