---
title: "data-juicer"
type: "tool"
slug: "datajuicer-data-juicer"
canonical_url: "https://www.graphcanon.com/tools/datajuicer-data-juicer"
github_url: "https://github.com/datajuicer/data-juicer"
homepage_url: "https://datajuicer.github.io/data-juicer/"
stars: 6671
forks: 392
primary_language: "Python"
license: "Apache-2.0"
categories: ["developer-tools", "model-training"]
tags: ["data-science", "data-visualization", "data-pipeline", "instruction-tuning", "data-analysis", "large-language-models", "foundation-models", "data-processing"]
updated_at: "2026-07-07T18:37:59.316015+00:00"
---

# data-juicer

> Data processing for foundation models

A comprehensive data processing framework designed to clean, synthesize, and analyze data for the development of large language models (LLMs) in a modular fashion.

## Facts

- Repository: https://github.com/datajuicer/data-juicer
- Homepage: https://datajuicer.github.io/data-juicer/
- Stars: 6,671 · Forks: 392 · Open issues: 66 · Watchers: 19
- Primary language: Python
- License: Apache-2.0
- Last pushed: 2026-07-07T09:15:12+00:00

## Categories

- [Developer Tools](/categories/developer-tools.md)
- [Model Training](/categories/model-training.md)

## Tags

data-science, data-visualization, data-pipeline, instruction-tuning, data-analysis, large language models, foundation-models, data-processing

## Related tools

- [ECC](/tools/affaan-m-ecc.md) - The agent harness performance optimization system (★ 226,962)
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT: Build, Deploy, and Run AI Agents (★ 185,417)
- [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019)
- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347)
- [JavaGuide](/tools/snailclimb-javaguide.md) - Snailclimb/JavaGuide: 面试 & 后端通用面试指南，覆盖计算机基础、数据库、分布式、高并发、系统设计与 AI 应用开发 (★ 156,863)
- [browser-use](/tools/browser-use-browser-use.md) - 🌐 Make websites accessible for AI agents. Automate tasks online with ease. (★ 103,315)
- [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,711)
- [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,150)

## README (excerpt)

```text
#  Data-Juicer: The Data Operating System for the Foundation Model Era
<p align="center">
  <a href="https://pypi.org/project/py-data-juicer"><img src="https://img.shields.io/pypi/v/py-data-juicer?logo=pypi&color=026cad" alt="PyPI"></a>
  <a href="https://pepy.tech/projects/py-data-juicer"><img src="https://static.pepy.tech/personalized-badge/py-data-juicer?period=total&units=INTERNATIONAL_SYSTEM&left_color=grey&right_color=green&left_text=downloads" alt="Downloads"></a>
   <a href="https://hub.docker.com/r/datajuicer/data-juicer"><img src="https://img.shields.io/docker/v/datajuicer/data-juicer?logo=docker&label=Docker&color=498bdf" alt="Docker"></a>
  <br>
  <a href="https://datajuicer.github.io/data-juicer/"><img src="https://img.shields.io/badge/📖_Docs-Website-026cad" alt="Docs"></a>
  <a href="https://datajuicer.github.io/data-juicer/en/main/docs/Operators.html"><img src="https://img.shields.io/badge/🧩_Operators-200+-blue" alt="Operators"></a>
  <a href="https://github.com/datajuicer/data-juicer-hub"><img src="https://img.shields.io/badge/🍳_Recipes-50+-brightgreen" alt="Recipes"></a>
  <br>
  <a href="https://datajuicer.github.io/data-juicer/zh_CN/main/index_ZH.html"><img src="https://img.shields.io/badge/🇨🇳_文档-主页-red" alt="Chinese"></a>
  <a href="https://arxiv.org/abs/2501.14755"><img src="https://img.shields.io/badge/NeurIPS'25_Spotlight-2.0-B31B1B?logo=arxiv" alt="Paper"></a>
  <a href="https://github.com/datajuicer/data-juicer">
    <img src="https://img.shields.io/endpoint?style=flat&url=https%3A%2F%2Fgist.githubusercontent.com%2FHYLcool%2Ff856b14416f08f73d05d32fd992a9c29%2Fraw%2Ftotal_cov.json&label=coverage&logo=codecov&color=4c1" alt="Coverage">
  </a>
</p>

<p align="center">
  <b>Multimodal | Cloud-Native | AI-Ready | Large-Scale </b>
</p>

Data-Juicer (DJ) transforms raw data chaos into AI-ready intelligence. It treats data processing as *composable infrastructure*—providing modular building blocks to clean, synthesize, and analyze data across the entire AI lifecycle, unlocking latent value in every byte.

Whether you're deduplicating web-scale pre-training corpora, curating agent interaction traces, or preparing domain-specific RAG indices, DJ scales seamlessly from your laptop to thousand-node clusters—no glue code required.

> **Alibaba Cloud PAI** has deeply integrated Data-Juicer into its data processing products.  See **[Quickly submit a DataJuicer job](https://www.alibabacloud.com/help/en/pai/user-guide/quickly-submit-a-datajuicer-task)**.

---

## 🚀 Quick Start

**Zero-install exploration**: 
- [JupyterLab Playground with Tutorials](http://8.138.149.181/) 
- [Ask DJ Copilot](https://datajuicer.github.io/data-juicer/en/main/docs_index.html)

**Install & run**:
```bash
uv pip install py-data-juicer
dj-process --config demos/process_simple/process.yaml
```

**Or compose in Python**:
```python
from data_juicer.core.data import NestedDataset
from data_juicer.ops.filter import TextLengthFilter
from data_juicer.ops.mapper import WhitespaceNormalizationMapper

ds = NestedDataset.from_dict({
    "text": ["Short", "This passes the filter.", "Text   with   spaces"]
})
res_ds = ds.process([
    TextLengthFilter(min_len=10),
    WhitespaceNormalizationMapper()
])

for s in res_ds:
    print(s)
```


---

## ✨ Why Data-Juicer?

### 1. Modular & Extensible Architecture
- **200+ operators** spanning text, image, audio, video, and multimodal data
- **Recipe-first**: Reproducible YAML pipelines you can version, share, and fork like code
- **Composable**: Drop in a single operator, chain complex workflows, or orchestrate full pipelines
- **Hot-reload**: Iterate on operators without pipeline restarts

### 2. Full-Spectrum Data Intelligence
- **Foundation Models**: Pre-training, fine-tuning, RL, and evaluation-grade curation
- **Agent Systems**: Clean tool traces, structure context, de-identification, and quality gating
- **RAG & Analytics**: Extraction, normalization, semantic chunking, deduplication, and data profil
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/datajuicer-data-juicer`](/api/graphcanon/tools/datajuicer-data-juicer)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_