{"data":{"slug":"datajuicer-data-juicer","name":"data-juicer","tagline":"Data processing for and with foundation models","github_url":"https://github.com/datajuicer/data-juicer","owner":"datajuicer","repo":"data-juicer","owner_avatar_url":"https://avatars.githubusercontent.com/u/223222708?v=4","primary_language":"Python","stars":6897,"forks":404,"topics":["data","data-analysis","data-pipeline","data-processing","data-science","data-visualization","foundation-models","instruction-tuning","large-language-models","llm","llms","multi-modal","pre-training","synthetic-data"],"archived":false,"github_pushed_at":"2026-08-13T09:19:31+00:00","maintenance_label":"Very active","stars_delta_30d":166,"url":"https://www.graphcanon.com/tools/datajuicer-data-juicer","markdown_url":"https://www.graphcanon.com/tools/datajuicer-data-juicer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/datajuicer-data-juicer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=datajuicer-data-juicer","description":"Data processing for and with foundation models!  🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷","homepage_url":"https://datajuicer.github.io/data-juicer/","license":"Apache-2.0","open_issues":59,"watchers":20,"ai_summary":"Library and tools for processing data used in the training of foundational AI models, including large language models (LLMs). Provides a pipeline for instruction tuning, synthetic data generation, and other preprocessing tasks necessary for model preparation.","readme_excerpt":"## 🚀 Quick Start\n\n**Zero-install exploration**:\n- [JupyterLab Playground with Tutorials](http://8.138.149.181/)\n- [Ask DJ Copilot](https://datajuicer.github.io/data-juicer/en/main/docs_index.html)\n\n**Install & run**:\n```bash\nuv pip install py-data-juicer\ndj-process --config demos/process_simple/process.yaml\n```\n\n**Or compose in Python**:\n```python\nfrom data_juicer.core.data import NestedDataset\nfrom data_juicer.ops.filter import TextLengthFilter\nfrom data_juicer.ops.mapper import WhitespaceNormalizationMapper\n\nds = NestedDataset.from_dict({\n    \"text\": [\"Short\", \"This passes the filter.\", \"Text   with   spaces\"]\n})\nres_ds = ds.process([\n    TextLengthFilter(min_len=10),\n    WhitespaceNormalizationMapper()\n])\n\nfor s in res_ds:\n    print(s)\n```\n\n\n---\n\n---\n\n## 📄 License & Attribution\n\nData-Juicer is released under the [Apache License 2.0](LICENSE).\nAttribution is appreciated: please use our [badge](https://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com/data_juicer/assets/DJ-Org-Logo.jpeg), or text as \"This project uses Data-Juicer: https://github.com/datajuicer\".\n\n---","github_created_at":"2023-08-01T09:16:41+00:00","created_at":"2026-07-07T17:34:41.414344+00:00","updated_at":"2026-08-17T12:00:59.499946+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"foundation-models","name":"foundation-models"},{"slug":"instruction-tuning","name":"instruction-tuning"},{"slug":"large-language-models","name":"large language models"},{"slug":"llm","name":"llm"},{"slug":"synthetic-data","name":"synthetic-data"}],"trust":{"provenance":{"is_fork":false,"github_id":673277958,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-17T12:00:58.565Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":4,"days_since_push":4,"last_release_at":"2026-08-07T08:09:20Z","stars_delta_30d":166,"open_issues_delta_30d":-3},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:03:40.786Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-17T12:00:59.135Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-17T12:00:59.135Z","managed_saas":false},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-17T12:00:59.135Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-17T12:00:59.135Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-17T12:00:59.135Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-17T12:00:59.135Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to preprocess large datasets specifically for training large language models (LLMs) with pipelines that support sophisticated processes like instruction tuning."],"when_not_to_use":["If your project does not involve foundational AI model training or if you do not require advanced data processing capabilities such as synthetic data generation."],"source":"enrich:decision_facts","observed_at":"2026-07-12T13:29:30.470Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"A Python library for foundational AI model data processing, offering a pipeline for tasks like instruction tuning and synthetic data generation."}]}}