{"data":{"slug":"huggingface-tokenizers","name":"tokenizers","tagline":"Fast State-of-the-Art Tokenizers optimized for Research and Production","github_url":"https://github.com/huggingface/tokenizers","owner":"huggingface","repo":"tokenizers","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Rust","stars":11047,"forks":1201,"topics":["bert","gpt","language-model","natural-language-processing","natural-language-understanding","nlp","transformers"],"archived":false,"github_pushed_at":"2026-09-18T08:07:38+00:00","maintenance_label":"Very active","stars_delta_30d":107,"url":"https://www.graphcanon.com/tools/huggingface-tokenizers","markdown_url":"https://www.graphcanon.com/tools/huggingface-tokenizers.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-tokenizers","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-tokenizers","description":"💥 Fast State-of-the-Art Tokenizers optimized for Research and Production","homepage_url":"https://huggingface.co/docs/tokenizers","license":"Apache-2.0","open_issues":259,"watchers":122,"ai_summary":"A library for fast tokenization, crucial for natural language processing tasks, supporting models like BERT and GPT.","readme_excerpt":"## Installation\n\nYou can install from source using:\n```bash\npip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python\n```\n\nor install the released versions with\n\n```bash\npip install tokenizers\n```","github_created_at":"2019-11-01T17:52:20+00:00","created_at":"2026-07-11T23:11:44.440063+00:00","updated_at":"2026-09-20T04:29:50.353896+00:00","categories":[{"slug":"developer-tools","name":"Developer Tools","url":"https://www.graphcanon.com/categories/developer-tools","markdown_url":"https://www.graphcanon.com/categories/developer-tools.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/developer-tools"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"bert","name":"bert"},{"slug":"gpt","name":"gpt"},{"slug":"language-model","name":"language-model"},{"slug":"natural-language-processing","name":"natural language processing"},{"slug":"natural-language-understanding","name":"natural-language-understanding"},{"slug":"nlp","name":"nlp"},{"slug":"transformers","name":"transformers"}],"trust":{"provenance":{"is_fork":false,"github_id":219035799,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:18:58.419Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":1,"days_since_push":0,"last_release_at":"2026-09-03T10:09:56Z","stars_delta_30d":107,"open_issues_delta_30d":-4},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:19:00.914Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:18:57.323Z"},"languages":{"value":["rust"],"source":"github.language","observed_at":"2026-09-18T08:18:57.323Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:18:57.323Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need high-speed tokenization for NLP tasks, especially with models like BERT and GPT.","For projects that require integration with Rust for performance optimization.","If you are working in a production environment where speed and efficiency are critical.","When you need a library that is actively maintained and supports the latest NLP models."],"when_not_to_use":["If your project is strictly Python-based and does not require the performance benefits of Rust.","When you are working with less common NLP models that are not supported by this library.","If you prefer a more generalized NLP library that does not specialize in tokenization.","When you need a solution that is easier to install and configure without the need for specific bindings."],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:25:10.209Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Tokenizers is a Rust-based library for fast tokenization, optimized for both research and production environments, supporting models like BERT and GPT."}]}}