{"data":{"slug":"huggingface-datatrove","name":"datatrove","tagline":"Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.","github_url":"https://github.com/huggingface/datatrove","owner":"huggingface","repo":"datatrove","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Python","stars":3153,"forks":279,"topics":[],"archived":false,"github_pushed_at":"2026-07-03T11:31:05+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/huggingface-datatrove","markdown_url":"https://www.graphcanon.com/tools/huggingface-datatrove.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-datatrove","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-datatrove","description":"Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.","homepage_url":null,"license":"Apache-2.0","open_issues":92,"watchers":48,"ai_summary":null,"readme_excerpt":"## Installation\n\nRequires Python 3.10+.\n\n```bash\nuv sync\n```\nAvailable flavours (combine them by repeating `--extra`, e.g. `uv sync --extra processing --extra s3`):\n- `all` installs everything: `uv sync --extra all`\n- `io` dependencies to read `warc/arc/wet` files and arrow/parquet/[Optimized-parquet](https://huggingface.co/docs/hub/en/datasets-libraries#optimized-parquet-files) formats: `uv sync --extra io`\n- `processing` dependencies for text extraction, filtering and tokenization: `uv sync --extra processing`\n- `s3` s3 support: `uv sync --extra s3`\n- `cli` for command line tools: `uv sync --extra cli`\n- `ray` for distributed compute engine: `uv sync --extra ray`\n- `inference` for LLM inference pipelines: `uv sync --extra inference`\n- `decont` for decontamination with lighteval: `uv sync --extra decont`\n- `multilingual` for multilingual text processing: `uv sync --extra multilingual`","github_created_at":"2023-06-14T12:05:28+00:00","created_at":"2026-07-11T10:34:15.28882+00:00","updated_at":"2026-07-12T03:46:29.383209+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"developer-tools","name":"Developer Tools","url":"https://www.graphcanon.com/categories/developer-tools","markdown_url":"https://www.graphcanon.com/categories/developer-tools.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/developer-tools"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":653623369,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:34:15.976Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":0,"days_since_push":7,"last_release_at":"2026-03-04T13:50:46Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:17.012Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:34:16.580Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-11T10:34:16.580Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T10:34:16.580Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T10:34:16.580Z"}}}}