{"data":{"slug":"bheinzerling-bpemb","name":"bpemb","tagline":"Pre-trained subword embeddings in 275 languages using Byte-Pair Encoding","github_url":"https://github.com/bheinzerling/bpemb","owner":"bheinzerling","repo":"bpemb","owner_avatar_url":"https://avatars.githubusercontent.com/u/4348795?v=4","primary_language":"Python","stars":1224,"forks":100,"topics":["embeddings","multilingual","natural-language-processing","nlp","subword-embeddings"],"archived":false,"github_pushed_at":"2024-10-01T02:49:47+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/bheinzerling-bpemb","markdown_url":"https://www.graphcanon.com/tools/bheinzerling-bpemb.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bheinzerling-bpemb","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bheinzerling-bpemb","description":"Pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE)","homepage_url":"https://nlp.h-its.org/bpemb","license":"MIT","open_issues":6,"watchers":23,"ai_summary":"A repository containing pre-trained subword embeddings for multiple languages using the Byte-Pair Encoding method, useful for natural language processing tasks.","readme_excerpt":"# BPEmb\n\nBPEmb is a collection of pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) and trained on Wikipedia. Its intended use is as input for neural models in natural language processing.\n\n[Website](https://nlp.h-its.org/bpemb) ・ \n[Usage](#usage) ・ \n[Download](#downloads-for-each-language) ・ \n[MultiBPEmb](#multibpemb) ・ \n[Paper (pdf)](http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf) ・ \n[Citing BPEmb](#citing-bpemb)\n\n\n\n## Usage\n\nInstall BPEmb with pip:\n\n```bash\npip install bpemb\n```\n\nEmbeddings and SentencePiece models will be downloaded automatically the first time you use them.\n\n```python\n>>> from bpemb import BPEmb\n# load English BPEmb model with default vocabulary size (10k) and 50-dimensional embeddings\n>>> bpemb_en = BPEmb(lang=\"en\", dim=50)\ndownloading https://nlp.h-its.org/bpemb/en/en.wiki.bpe.vs10000.model\ndownloading https://nlp.h-its.org/bpemb/en/en.wiki.bpe.vs10000.d50.w2v.bin.tar.gz\n```\n\nYou can do two main things with BPEmb. The first is subword segmentation:\n```python\n# apply English BPE subword segmentation model\n>>> bpemb_en.encode(\"Stratford\")\n['▁strat', 'ford']\n# load Chinese BPEmb model with vocabulary size 100k and default (100-dim) embeddings\n>>> bpemb_zh = BPEmb(lang=\"zh\", vs=100000)\n# apply Chinese BPE subword segmentation model\n>>> bpemb_zh.encode(\"这是一个中文句子\")  # \"This is a Chinese sentence.\"\n['▁这是一个', '中文', '句子']  # [\"This is a\", \"Chinese\", \"sentence\"]\n```\n\nIf / how a word gets split depends on the vocabulary size. Generally, a smaller vocabulary size will yield a segmentation into many subwords, while a large vocabulary size will result in frequent words not being split:\n\n| vocabulary size | segmentation |\n| --- | --- |\n| 1000 | ['▁str', 'at', 'f', 'ord'] |\n| 3000 |  ['▁str', 'at', 'ford'] |\n| 5000 | ['▁str', 'at', 'ford'] |\n| 10000 | ['▁strat', 'ford'] |\n| 25000 | ['▁stratford'] |\n| 50000 | ['▁stratford'] |\n| 100000 | ['▁stratford'] |\n| 200000 | ['▁stratford'] |\n\n\nThe second purpose of BPEmb is to provide pretrained subword embeddings:\n\n```python\n# Embeddings are wrapped in a gensim KeyedVectors object\n>>> type(bpemb_zh.emb)\ngensim.models.keyedvectors.Word2VecKeyedVectors\n# You can use BPEmb objects like gensim KeyedVectors\n>>> bpemb_en.most_similar(\"ford\")\n[('bury', 0.8745079040527344),\n ('ton', 0.8725000619888306),\n ('well', 0.871537446975708),\n ('ston', 0.8701574206352234),\n ('worth', 0.8672043085098267),\n ('field', 0.859795331954956),\n ('ley', 0.8591548204421997),\n ('ington', 0.8126075267791748),\n ('bridge', 0.8099068999290466),\n ('brook', 0.7979353070259094)]\n>>> type(bpemb_en.vectors)\nnumpy.ndarray\n>>> bpemb_en.vectors.shape\n(10000, 50)\n>>> bpemb_zh.vectors.shape\n(100000, 100)\n```\n\nTo use subword embeddings in your neural network, either encode your input into subword IDs:\n```python\n>>> ids = bpemb_zh.encode_ids(\"这是一个中文句子\")\n[25950, 695, 20199]\n>>> bpemb_zh.vectors[ids].shape\n(3, 100)\n```\n\nOr use the `embed` method:\n```python\n# apply Chinese subword segmentation and perform embedding lookup\n>>> bpemb_zh.embed(\"这是一个中文句子\").shape\n(3, 100)\n```\n\n# Downloads for each language\n\n[ab (Abkhazian)](http://nlp.h-its.org/bpemb/ab) ・ \n[ace (Achinese)](http://nlp.h-its.org/bpemb/ace) ・ \n[ady (Adyghe)](http://nlp.h-its.org/bpemb/ady) ・ \n[af (Afrikaans)](http://nlp.h-its.org/bpemb/af) ・ \n[ak (Akan)](http://nlp.h-its.org/bpemb/ak) ・ \n[als (Alemannic)](http://nlp.h-its.org/bpemb/als) ・ \n[am (Amharic)](http://nlp.h-its.org/bpemb/am) ・ \n[an (Aragonese)](http://nlp.h-its.org/bpemb/an) ・ \n[ang (Old English)](http://nlp.h-its.org/bpemb/ang) ・ \n[ar (Arabic)](http://nlp.h-its.org/bpemb/ar) ・ \n[arc (Official Aramaic)](http://nlp.h-its.org/bpemb/arc) ・ \n[arz (Egyptian Arabic)](http://nlp.h-its.org/bpemb/arz) ・ \n[as (Assamese)](http://nlp.h-its.org/bpemb/as) ・ \n[ast (Asturian)](http://nlp.h-its.org/bpemb/ast) ・ \n[atj (Atikamekw)](http://nlp.h-its.org/bpemb/atj) ・ \n[av (Avaric)](http://nlp.h-its.org/bpemb/av) ・ \n[ay (Aymara)](http://nlp.h-its.org/bpemb/ay) ・ \n[az (Azerbaijani)](htt","github_created_at":"2017-10-04T13:03:23+00:00","created_at":"2026-07-11T11:31:12.974046+00:00","updated_at":"2026-08-22T12:01:32.780695+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"}],"tags":[{"slug":"embeddings","name":"embeddings"},{"slug":"multilingual","name":"multilingual"},{"slug":"natural-language-processing","name":"natural-language-processing"},{"slug":"nlp","name":"nlp"},{"slug":"subword-embeddings","name":"subword-embeddings"}],"trust":{"provenance":{"is_fork":false,"github_id":105769029,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-22T12:01:32.069Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":690,"last_release_at":null,"stars_delta_30d":2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:31:14.326Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-22T12:01:32.500Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-22T12:01:32.500Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-22T12:01:32.500Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires Python environment to operate effectively across various multilingual applications"]},"constraints":null,"when_to_use":["When working on multilingual projects that span a vast array of languages (up to 275) where language-specific data is sparse or unavailable","For NLP tasks requiring efficient handling of low-frequency words and rare terms across different languages"],"when_not_to_use":["If your project focuses solely on high-resource languages like English, Spanish, French where more specialized models provide better performance per task","When the task specifically requires character-level or word-level embeddings and not subword tokenization provided by Byte-Pair Encoding (BPE)"],"source":"enrich:decision_facts","observed_at":"2026-07-12T18:50:41.348Z"},"constraint_facets":null,"decision_summary":[{"label":"Requirements","value":"Requires Python environment to operate effectively across various multilingual applications"},{"label":"Adopt for","value":"bpemb provides pre-trained subword embeddings using Byte-Pair Encoding for up to 275 languages, which can be beneficial in multi-lingual NLP tasks."},{"label":"License detail","value":"MIT License: Permissive free software license granting users freedom to use, modify, and distribute the software."}]}}