tokenizers
Enrichment pending💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
GraphCanon updated today · GitHub synced today
11k
Stars
1.1k
Forks
226
Open issues
120
Watchers
today
Last push
Rust Apache-2.0Created Nov 1, 2019
Trust & integrity
Full report- Maintenance
- Very active (0d since push)
- As of today · Source: github_public_v1
- Provenance
- Not a fork · Organization account
- As of today · Source: github_public_v1
- Security (OSV)
- No lockfile
- As of today · Source: none
Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.
Backing
Company and funding context for Hugging Face. Display-only - not part of trust score or organic ranking.
- Company
- Hugging Face·GitHub org profile·today
- Employees
- 160·Wikidata (P1128 employees)·today
- Funding
- $235,000,000 (2023-08)·GraphCanon curated seed (public press)·today
- Commercial model
- OSS + managed cloud·GraphCanon curated seed·today
Overview
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Capability facts
- Languages
- rust
Source: github.language · Jul 11, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Python runtimePython
Source: README excerpt (regex_v1, Jul 11, 2026)
install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/pythonSource link
Tags
README
Installation
You can install from source using:
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
or install the released versions with
pip install tokenizers