{"data":{"slug":"huggingface-datatrove","name":"datatrove","tagline":"Platform-agnostic customizable pipeline processing blocks for data processing and transformation.","github_url":"https://github.com/huggingface/datatrove","owner":"huggingface","repo":"datatrove","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Python","stars":3250,"forks":288,"topics":[],"archived":false,"github_pushed_at":"2026-08-06T15:27:26+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/huggingface-datatrove","markdown_url":"https://www.graphcanon.com/tools/huggingface-datatrove.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-datatrove","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-datatrove","description":"Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.","homepage_url":null,"license":"Apache-2.0","open_issues":93,"watchers":49,"ai_summary":"Provides tools to create and manage pipelines for various data processing tasks like text extraction and tokenization, supporting different file formats and distributed computing options.","readme_excerpt":"## Installation\n\nRequires Python 3.10+.\n\n```bash\nuv sync\n```\nAvailable flavours (combine them by repeating `--extra`, e.g. `uv sync --extra processing --extra s3`):\n- `all` installs everything: `uv sync --extra all`\n- `io` dependencies to read `warc/arc/wet` files and arrow/parquet/[Optimized-parquet](https://huggingface.co/docs/hub/en/datasets-libraries#optimized-parquet-files) formats: `uv sync --extra io`\n- `processing` dependencies for text extraction, filtering and tokenization: `uv sync --extra processing`\n- `s3` s3 support: `uv sync --extra s3`\n- `cli` for command line tools: `uv sync --extra cli`\n- `ray` for distributed compute engine: `uv sync --extra ray`\n- `inference` for LLM inference pipelines: `uv sync --extra inference`\n- `decont` for decontamination with lighteval: `uv sync --extra decont`\n- `multilingual` for multilingual text processing: `uv sync --extra multilingual`","github_created_at":"2023-06-14T12:05:28+00:00","created_at":"2026-07-11T10:34:15.28882+00:00","updated_at":"2026-08-07T00:01:38.406369+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"data-processing","name":"data-processing"},{"slug":"distributed-computing","name":"distributed-computing"},{"slug":"file-formats-support","name":"file-formats-support"},{"slug":"pipelines","name":"pipelines"},{"slug":"text-tokenization","name":"text-tokenization"}],"trust":{"provenance":{"is_fork":false,"github_id":653623369,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-07T00:01:37.623Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":0,"last_release_at":"2026-03-04T13:50:46Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:17.012Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-07T00:01:38.068Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-07T00:01:38.068Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-07T00:01:38.068Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-07T00:01:38.068Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you require a flexible configuration that allows for custom pipelines, supporting text extraction, tokenization, and multilingual text processing.","If your project includes file handling like warc, arc, wet, arrow/parquet, or optimized-parquet formats with s3 support."],"when_not_to_use":["Avoid datatrove if you are not working within Python 3.10+, as it is not compatible with earlier versions.","Do not use if you require real-time data processing functionalities that go beyond the package's current capabilities, such as streaming data handling."],"source":"enrich:decision_facts","observed_at":"2026-07-15T09:38:51.688Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Datatrove is ideal for users needing platform-agnostic customizable pipeline blocks for data processing and transformation across various file formats with built-in support for distributed computing options."}]}}