GraphCanon updated 3d · GitHub synced 3d
Decision brief
A Python library for foundational AI model data processing, offering a pipeline for tasks like instruction tuning and synthetic data generation.
Good fit when
- When you need to preprocess large datasets specifically for training large language models (LLMs) with pipelines that support sophisticated processes like instruction tuning.
Avoid when
- If your project does not involve foundational AI model training or if you do not require advanced data processing capabilities such as synthetic data generation.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Adoption
Package downloads where a registry match exists. GitHub stars (6,897) are secondary evidence.
- Docker Hub pulls (30d)
- 14,309·Docker Hub API·3d
Maintenance and security
Full trust report- Maintenance
- Very active (4d since push)
- As of 3d
- Provenance
- Not a fork · Organization account
- As of 3d
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install data-juicerHow it fits your stack(11)
Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.
Integrates
Related
Relationship graph
Optional deeper exploration of typed edges and category neighbours.
Similar tools
Same-category neighbours not already linked as typed edges.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Library and tools for processing data used in the training of foundational AI models, including large language models (LLMs). Provides a pipeline for instruction tuning, synthetic data generation, and other preprocessing tasks necessary for model preparation.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 17, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 17, 2026
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 17, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 17, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 17, 2026)
**Or compose in Python**:Source link
Tags
README
🚀 Quick Start
Zero-install exploration:
Install & run:
uv pip install py-data-juicer
dj-process --config demos/process_simple/process.yaml
Or compose in Python:
from data_juicer.core.data import NestedDataset
from data_juicer.ops.filter import TextLengthFilter
from data_juicer.ops.mapper import WhitespaceNormalizationMapper
ds = NestedDataset.from_dict({
"text": ["Short", "This passes the filter.", "Text with spaces"]
})
res_ds = ds.process([
TextLengthFilter(min_len=10),
WhitespaceNormalizationMapper()
])
for s in res_ds:
print(s)
📄 License & Attribution
Data-Juicer is released under the Apache License 2.0. Attribution is appreciated: please use our badge, or text as "This project uses Data-Juicer: https://github.com/datajuicer".
For agents
This page has a .md twin and JSON over the API.