GraphCanon
Home/LLM Frameworks/datatrove
datatrove logo

datatrove

Enrichment pending
huggingface/datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

GitHub

GraphCanon updated today · GitHub synced today

3.2k
Stars
279
Forks
92
Open issues
48
Watchers
1w
Last push
Python Apache-2.0Created Jun 14, 2023

Trust & integrity

Full report
Maintenance
Active (7d since push)
As of today · Source: github_public_v1
Provenance
Not a fork · Organization account
As of today · Source: github_public_v1
Security (OSV)
No lockfile
As of today · Source: none

Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.

Backing

Company and funding context for Hugging Face. Display-only - not part of trust score or organic ranking.

Company
Hugging Face·GitHub org profile·today
Employees
160·Wikidata (P1128 employees)·today
Funding
$235,000,000 (2023-08)·GraphCanon curated seed (public press)·today
Commercial model
OSS + managed cloud·GraphCanon curated seed·today

Overview

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

Capability facts

CLI
CLI entrypoint

Source: pyproject.toml:[project.scripts] · Jul 11, 2026

Languages
python

Source: github.language+pyproject.toml · Jul 11, 2026

Categories

LLM FrameworksInference & ServingDeveloper Tools

Graph entities

Hugging Facecompany

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Jul 11, 2026)

Requires Python 3.10+.
Source link

Tags

python

README

Installation

Requires Python 3.10+.

uv sync

Available flavours (combine them by repeating --extra, e.g. uv sync --extra processing --extra s3):

  • all installs everything: uv sync --extra all
  • io dependencies to read warc/arc/wet files and arrow/parquet/Optimized-parquet formats: uv sync --extra io
  • processing dependencies for text extraction, filtering and tokenization: uv sync --extra processing
  • s3 s3 support: uv sync --extra s3
  • cli for command line tools: uv sync --extra cli
  • ray for distributed compute engine: uv sync --extra ray
  • inference for LLM inference pipelines: uv sync --extra inference
  • decont for decontamination with lighteval: uv sync --extra decont
  • multilingual for multilingual text processing: uv sync --extra multilingual