Home/Data & Retrieval/datatrove

datatrove

huggingface/datatrove

Platform-agnostic customizable pipeline processing blocks for data processing and transformation.

GitHub

GraphCanon updated Oct 12, 2026 · GitHub synced Oct 12, 2026

124views this month

3.4k stars312 forksLast push Oct 6, 2026 Python Apache-2.0

Decision brief

Datatrove is ideal for users needing platform-agnostic customizable pipeline blocks for data processing and transformation across various file formats with built-in support for distributed computing options.

Good fit when

When you require a flexible configuration that allows for custom pipelines, supporting text extraction, tokenization, and multilingual text processing.
If your project includes file handling like warc, arc, wet, arrow/parquet, or optimized-parquet formats with s3 support.

Avoid when

Avoid datatrove if you are not working within Python 3.10+, as it is not compatible with earlier versions.
Do not use if you require real-time data processing functionalities that go beyond the package's current capabilities, such as streaming data handling.

Observed Jul 15, 2026 · Source: enrich:decision_facts

datatrove alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Very active (5d since push); As of Oct 12, 2026
Provenance: Not a fork · Organization account; As of Oct 12, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Backing

Company context for Hugging Face. Display-only - separate from trust and ranking.

Company: Hugging Face·GitHub org profile·Jul 11, 2026
Employees: 160·Wikidata (P1128 employees)·Jul 11, 2026
Funding: $235,000,000 (2023-08)·GraphCanon curated seed (public press)·Jul 11, 2026
Commercial model: OSS + managed cloud·GraphCanon curated seed·Jul 11, 2026

Install

pip install datatrove

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

llm-app

Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data

59k★

opendataloader-pdf

PDF Parser for AI-ready data

29k★

datasets

Largest hub of ready-to-use datasets for AI models

22k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Provides tools to create and manage pipelines for various data processing tasks like text extraction and tokenization, supporting different file formats and distributed computing options.

Capability facts

CLI: CLI entrypoint
Source: pyproject.toml:[project.scripts] · Oct 12, 2026
Languages: python
Source: github.language+pyproject.toml · Oct 12, 2026

Graph entities

Hugging Facecompany

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Oct 12, 2026)

Requires Python 3.10+.

Source link

README

Installation Requires Python 3.10+. Available flavours (combine them by repeating , e.g. ): installs everything: dependencies to read files and arrow/parquet/Optimized parquet formats: dependencies for text extraction, filtering and tokenization: s3 support: for command line tool...

For agents

This page has a .md twin and JSON over the API.

/tools/huggingface-datatrove.md GET /api/graphcanon/tools/huggingface-datatrove GET /api/graphcanon/graph?tool=huggingface-datatrove