datatrove
Platform-agnostic customizable pipeline processing blocks for data processing and transformation.
GraphCanon updated 2w · GitHub synced 2w · 28 views this month
Decision brief
Datatrove is ideal for users needing platform-agnostic customizable pipeline blocks for data processing and transformation across various file formats with built-in support for distributed computing options.
Good fit when
- When you require a flexible configuration that allows for custom pipelines, supporting text extraction, tokenization, and multilingual text processing.
- If your project includes file handling like warc, arc, wet, arrow/parquet, or optimized-parquet formats with s3 support.
Avoid when
- Avoid datatrove if you are not working within Python 3.10+, as it is not compatible with earlier versions.
- Do not use if you require real-time data processing functionalities that go beyond the package's current capabilities, such as streaming data handling.
Observed Jul 15, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Very active (0d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for Hugging Face. Display-only - separate from trust and ranking.
- Company
- Hugging Face·GitHub org profile·1mo
- Employees
- 160·Wikidata (P1128 employees)·1mo
- Funding
- $235,000,000 (2023-08)·GraphCanon curated seed (public press)·1mo
- Commercial model
- OSS + managed cloud·GraphCanon curated seed·1mo
Install
pip install datatroveSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides tools to create and manage pipelines for various data processing tasks like text extraction and tokenization, supporting different file formats and distributed computing options.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 7, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 7, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Tags
README
Installation
Requires Python 3.10+.
uv sync
Available flavours (combine them by repeating --extra, e.g. uv sync --extra processing --extra s3):
allinstalls everything:uv sync --extra alliodependencies to readwarc/arc/wetfiles and arrow/parquet/Optimized-parquet formats:uv sync --extra ioprocessingdependencies for text extraction, filtering and tokenization:uv sync --extra processings3s3 support:uv sync --extra s3clifor command line tools:uv sync --extra clirayfor distributed compute engine:uv sync --extra rayinferencefor LLM inference pipelines:uv sync --extra inferencedecontfor decontamination with lighteval:uv sync --extra decontmultilingualfor multilingual text processing:uv sync --extra multilingual
For agents
This page has a .md twin and JSON over the API.