{"data":{"slug":"huggingface-tokenizers","name":"tokenizers","tagline":"💥 Fast State-of-the-Art Tokenizers optimized for Research and Production","github_url":"https://github.com/huggingface/tokenizers","owner":"huggingface","repo":"tokenizers","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Rust","stars":10878,"forks":1140,"topics":["bert","gpt","language-model","natural-language-processing","natural-language-understanding","nlp","transformers"],"archived":false,"github_pushed_at":"2026-07-11T13:38:25+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/huggingface-tokenizers","markdown_url":"https://www.graphcanon.com/tools/huggingface-tokenizers.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-tokenizers","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-tokenizers","description":"💥 Fast State-of-the-Art Tokenizers optimized for Research and Production","homepage_url":"https://huggingface.co/docs/tokenizers","license":"Apache-2.0","open_issues":226,"watchers":120,"ai_summary":"A library of fast and efficient state-of-the-art tokenizers, vital for tasks in natural language processing, including training models like BERT and GPT.","readme_excerpt":"## Installation\n\nYou can install from source using:\n```bash\npip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python\n```\n\nor install the released versions with\n\n```bash\npip install tokenizers\n```","github_created_at":"2019-11-01T17:52:20+00:00","created_at":"2026-07-11T23:11:44.440063+00:00","updated_at":"2026-07-12T05:08:08.53919+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"bert","name":"bert"},{"slug":"nlp","name":"nlp"},{"slug":"natural-language-processing","name":"natural-language-processing"},{"slug":"gpt","name":"gpt"},{"slug":"natural-language-understanding","name":"natural-language-understanding"},{"slug":"transformers","name":"transformers"},{"slug":"language-model","name":"language-model"}],"trust":{"provenance":{"is_fork":false,"github_id":219035799,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:11:45.605Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":1,"days_since_push":0,"last_release_at":"2026-04-27T14:38:35Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:11:46.356Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T05:07:39.959Z"},"languages":{"value":["rust"],"source":"github.language","observed_at":"2026-07-12T05:07:39.959Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-12T05:07:39.959Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium"},"requirements":{"notes":["Installation can be done directly via pip or from source, offering flexibility for different project needs."],"min_ram_gb":4,"requires_docker":false},"constraints":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":false},"when_to_use":["When you require a library that is optimized both for research and production environments, ensuring efficiency in NLP tasks.","For projects involving state-of-the-art models like BERT or GPT where the performance of tokenization can have a significant impact on model training speed."],"when_not_to_use":["If your project is limited to older NLP models which do not require such advanced tokenizers, opting for something simpler might be more appropriate.","In scenarios where Rust-based tooling does not fit within your existing tech stack and there's no immediate plan or capability to integrate new languages."],"source":"enrich:decision_facts","observed_at":"2026-07-12T05:08:08.332Z"},"constraint_facets":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium"},{"label":"Requirements","value":"Min 4 GB RAM; Installation can be done directly via pip or from source, offering flexibility for different project needs."},{"label":"Adopt for","value":"Factual criteria for evaluating 'tokenizers'."},{"label":"License detail","value":"Apache-2.0"}]}}