Data-Juicer: The Data Operating System for the Foundation Model Era

Multimodal | Cloud-Native | AI-Ready | Large-Scale

Data-Juicer (DJ) transforms raw data chaos into AI-ready intelligence. It treats data processing as composable infrastructure—providing modular building blocks to clean, synthesize, and analyze data across the entire AI lifecycle, unlocking latent value in every byte.

Whether you're deduplicating web-scale pre-training corpora, curating agent interaction traces, or preparing domain-specific RAG indices, DJ scales seamlessly from your laptop to thousand-node clusters—no glue code required.

Alibaba Cloud PAI has deeply integrated Data-Juicer into its data processing products. See Quickly submit a DataJuicer job.

---

🚀 Quick Start

Zero-install exploration:

• JupyterLab Playground with Tutorials
• Ask DJ Copilot

Install & run:

uv pip install py-data-juicer
dj-process --config demos/process_simple/process.yaml

Or compose in Python:

from data_juicer.core.data import NestedDataset
from data_juicer.ops.filter import TextLengthFilter
from data_juicer.ops.mapper import WhitespaceNormalizationMapper

ds = NestedDataset.from_dict({
    "text": ["Short", "This passes the filter.", "Text   with   spaces"]
})
res_ds = ds.process([
    TextLengthFilter(min_len=10),
    WhitespaceNormalizationMapper()
])

for s in res_ds:
    print(s)

---

✨ Why Data-Juicer?

1. Modular & Extensible Architecture

• 200+ operators spanning text, image, audio, video, and multimodal data
• Recipe-first: Reproducible YAML pipelines you can version, share, and fork like code
• Composable: Drop in a single operator, chain complex workflows, or orchestrate full pipelines
• Hot-reload: Iterate on operators without pipeline restarts

2. Full-Spectrum Data Intelligence

• Foundation Models: Pre-training, fine-tuning, RL, and evaluation-grade curation
• Agent Systems: Clean tool traces, structure context, de-identification, and quality gating
• RAG & Analytics: Extraction, normalization, semantic chunking, deduplication, and data profil