{"data":{"slug":"data-prep-kit-data-prep-kit","name":"data-prep-kit","tagline":"Open source project for data preparation for GenAI applications","github_url":"https://github.com/data-prep-kit/data-prep-kit","owner":"data-prep-kit","repo":"data-prep-kit","owner_avatar_url":"https://avatars.githubusercontent.com/u/202559084?v=4","primary_language":"HTML","stars":952,"forks":253,"topics":["code-quality","data","data-prep","data-preparation","data-preprocessing","data-preprocessing-pipelines","datacuration","datarecipes","deduplication","finetuning","large-language-models","large-scale-data-processing","llm","llmapps","malware","python","ray","spark"],"archived":false,"github_pushed_at":"2026-07-14T16:29:06+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/data-prep-kit-data-prep-kit","markdown_url":"https://www.graphcanon.com/tools/data-prep-kit-data-prep-kit.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/data-prep-kit-data-prep-kit","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=data-prep-kit-data-prep-kit","description":"Open source project for data preparation for GenAI applications","homepage_url":"https://data-prep-kit.github.io/data-prep-kit/","license":"Apache-2.0","open_issues":223,"watchers":18,"ai_summary":"Data Prep Kit is designed to provide tools and pipelines for preparing data for large language models (LLM) and other GenAI applications, including deduplication, fine-tuning, and large-scale processing.","readme_excerpt":"## Installation\n\nThe latest version of the Data Prep Kit is available on PyPi for Python 3.10, 3.11, 3.12, and 3.13. It can be installed using: \n\n```bash\npip install uv\nuv pip install 'data-prep-toolkit-transforms[all]'\n```\n\nThis will install all available transforms. \n\nFor guidance on creating the virtual environment for installing the data prep kit, click [here](doc/quick-start/quick-start.md).\n\n---\n\n## &#x1F680; Getting Started <a name = \"gettingstarted\"></a>\n\n---\n\n## License\n\nAll source files must include a Copyright and License header. If you would like to see the detailed LICENSE click [here](LICENSE).","github_created_at":"2024-04-08T23:43:52+00:00","created_at":"2026-07-11T10:34:11.147925+00:00","updated_at":"2026-08-07T00:01:35.953621+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"code-quality","name":"code-quality"},{"slug":"data-prep","name":"data-prep"},{"slug":"data-preparation","name":"data-preparation"},{"slug":"data-preprocessing-pipelines","name":"data-preprocessing-pipelines"},{"slug":"deduplication","name":"deduplication"},{"slug":"finetuning","name":"finetuning"},{"slug":"large-language-models","name":"large language models"},{"slug":"llmapps","name":"llmapps"}],"trust":{"provenance":{"is_fork":false,"github_id":783968898,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-07T00:01:35.089Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":1,"days_since_push":23,"last_release_at":"2026-06-22T16:30:46Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:34:12.687Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-07T00:01:35.671Z"},"languages":{"value":["html"],"source":"github.language","observed_at":"2026-08-07T00:01:35.671Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-07T00:01:35.671Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Installation requires Python versions from 3.10 to 3.13."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"requires_docker":false},"when_to_use":["Use data-prep-kit when you are working with large language models (LLMs) or other GenAI applications and need comprehensive tools for data preparation, including deduplication and fine-tuning.","Consider it if your project involves handling large-scale data processing."],"when_not_to_use":["Avoid data-prep-kit if the project does not require Python compatibility or if Python versions earlier than 3.10 are in use since this toolkit supports only from Python 3.10 to 3.13.","Do not use it for tasks unrelated to GenAI applications as its specific features may not be beneficial."],"source":"enrich:decision_facts","observed_at":"2026-07-12T14:01:00.473Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Installation requires Python versions from 3.10 to 3.13."},{"label":"Adopt for","value":"Curated decision-critical facts for the tool 'data-prep-kit'."},{"label":"License detail","value":"Apache-2.0 license allows users to freely modify and distribute the software, provided that all copyright and permission notices are kept intact."}]}}