Home/Developer Tools/tokenizers

tokenizers

huggingface/tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

54views this month

11k stars1.2k forksLast push Sep 18, 2026 Rust Apache-2.0

Decision brief

Tokenizers is a Rust-based library for fast tokenization, optimized for both research and production environments, supporting models like BERT and GPT.

Good fit when

When you need high-speed tokenization for NLP tasks, especially with models like BERT and GPT.
For projects that require integration with Rust for performance optimization.

Avoid when

If your project is strictly Python-based and does not require the performance benefits of Rust.
When you are working with less common NLP models that are not supported by this library.

Observed Sep 18, 2026 · Source: enrich:decision_facts

tokenizers alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Very active (0d since push); As of Sep 18, 2026
Provenance: Not a fork · Organization account; As of Sep 18, 2026
Security (OSV): No lockfile; As of Sep 18, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Backing

Company context for Hugging Face. Display-only - separate from trust and ranking.

Company: Hugging Face·GitHub org profile·Jul 11, 2026
Employees: 160·Wikidata (P1128 employees)·Jul 11, 2026
Funding: $235,000,000 (2023-08)·GraphCanon curated seed (public press)·Jul 11, 2026
Commercial model: OSS + managed cloud·GraphCanon curated seed·Jul 11, 2026