data-juicer logo

data-juicer

datajuicer/data-juicer

Data processing for and with foundation models

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

151views this month

6.9k stars404 forksLast push Aug 13, 2026 Python Apache-2.0

Decision brief

A Python library for foundational AI model data processing, offering a pipeline for tasks like instruction tuning and synthetic data generation.

Good fit when

  • When you need to preprocess large datasets specifically for training large language models (LLMs) with pipelines that support sophisticated processes like instruction tuning.

Avoid when

  • If your project does not involve foundational AI model training or if you do not require advanced data processing capabilities such as synthetic data generation.

Observed Jul 12, 2026 · Source: enrich:decision_facts

Verify the decision

Adoption

Package downloads where a registry match exists. GitHub stars (6,897) are secondary evidence.

Docker Hub pulls (30d)
14,309·Docker Hub API·Aug 17, 2026

Maintenance and security

Full trust report
Maintenance
Very active (4d since push)
As of Aug 17, 2026
Provenance
Not a fork · Organization account
As of Aug 17, 2026
Security (OSV)
No lockfile
As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install data-juicer
PyPI

How it fits your stack(11)

Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.

Integrates

Relationship graph

Optional deeper exploration of typed edges and category neighbours.

Similar tools

Same-category neighbours not already linked as typed edges.

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Library and tools for processing data used in the training of foundational AI models, including large language models (LLMs). Provides a pipeline for instruction tuning, synthetic data generation, and other preprocessing tasks necessary for model preparation.

Capability facts

Deploy
Self-host

Source: dockerfile:Dockerfile · Aug 17, 2026

Docker
Dockerfile present

Source: dockerfile:Dockerfile · Aug 17, 2026

CLI
CLI entrypoint

Source: pyproject.toml:[project.scripts] · Aug 17, 2026

Languages
python

Source: github.language+pyproject.toml · Aug 17, 2026

Categories

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Aug 17, 2026)

**Or compose in Python**:
Source link

Tags

README

🚀 Quick Start Zero install exploration : JupyterLab Playground with Tutorials Ask DJ Copilot Install & run : Or compose in Python : 📄 License & Attribution Data Juicer is released under the Apache License 2.0. Attribution is appreciated: please use our badge, or text as "This p...

For agents

This page has a .md twin and JSON over the API.

Was this helpful?

Anonymous feedback helps us improve pages and translations.