{"data":{"slug":"nvidia-nemo-curator","name":"Curator","tagline":"Scalable data pre-processing and curation toolkit for LLMs","github_url":"https://github.com/NVIDIA-NeMo/Curator","owner":"NVIDIA-NeMo","repo":"Curator","owner_avatar_url":"https://avatars.githubusercontent.com/u/213689629?v=4","primary_language":"Python","stars":1731,"forks":320,"topics":["data","data-curation","data-prep","data-preparation","data-processing","data-processing-pipelines","data-quality","datacuration","datarecipes","deduplication","fast-data-processing","fine-tuning","large-language-models","large-scale-data-processing","llm","llm-data-quality","llmapps","python","semantic-deduplication"],"archived":false,"github_pushed_at":"2026-08-21T20:19:34+00:00","maintenance_label":"Very active","stars_delta_30d":50,"url":"https://www.graphcanon.com/tools/nvidia-nemo-curator","markdown_url":"https://www.graphcanon.com/tools/nvidia-nemo-curator.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nvidia-nemo-curator","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nvidia-nemo-curator","description":"Scalable data pre processing and curation toolkit for LLMs","homepage_url":null,"license":"Apache-2.0","open_issues":280,"watchers":18,"ai_summary":"NVIDIA-NeMo/Curator is a Python-based repository offering tools for scalable data processing and curation specifically designed for Large Language Models (LLMs). It includes functionalities such as deduplication, quality checks, and preparation pipelines which are crucial steps before the training phase of AI models.","readme_excerpt":"## Quick Start\n\nThree paths, depending on what you're trying to do. Each path is self-contained.\n\nNeMo Curator uses [`uv`](https://docs.astral.sh/uv/) for installation. Install it once:\n\n```bash\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\n---\n\n### Path C — Docker (recommended for video and audio)\n\nVideo and audio pipelines depend on system codec libraries; the published container ships them preconfigured.\n\n- Container: [nemo-curator on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator)\n- Setup instructions: [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/installation)\n\n**Full setup for all paths:** [Installation Guide](https://docs.nvidia.com/nemo/curator/latest/get-started/installation) • [Tutorials](tutorials/)\n\n---\n\n---\n\n## Installation and Deployment\n\n| Resource | Link |\n|----------|------|\n| Installation guide (CPU, GPU, Docker, source) | [docs.nvidia.com/nemo/curator/latest/get-started/installation](https://docs.nvidia.com/nemo/curator/latest/get-started/installation) |\n| Container image | [nemo-curator on NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-curator) |\n| Infrastructure (Slurm, Kubernetes, multi-node) | [Infrastructure docs](https://docs.nvidia.com/nemo/curator/latest/reference/infra) |\n| API reference | [API docs](https://docs.nvidia.com/nemo/curator/latest/api/reference/api-reference) |\n| Concepts | [Concepts](https://docs.nvidia.com/nemo/curator/latest/about/concepts) |\n\nSupported Python and dependency versions are defined in [`pyproject.toml`](pyproject.toml); the Python versions are also shown on the PyPI badge above. The README does not duplicate them to avoid drift.\n\n---\n\n---\n\n## License\n\nNeMo Curator is released under the **Apache License 2.0**. See [LICENSE](LICENSE) for the full text.","github_created_at":"2024-03-14T20:41:51+00:00","created_at":"2026-07-11T11:39:02.079984+00:00","updated_at":"2026-08-24T00:02:13.730309+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"curation-toolkit","name":"curation toolkit"},{"slug":"data-pre-processing","name":"data pre-processing"},{"slug":"deduplication","name":"deduplication"},{"slug":"llms","name":"llms"},{"slug":"preparation-pipelines","name":"preparation pipelines"},{"slug":"quality-checks","name":"quality checks"},{"slug":"scalable-data-processing","name":"scalable data processing"}],"trust":{"provenance":{"is_fork":false,"github_id":772255271,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-24T00:02:12.927Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":1,"days_since_push":2,"last_release_at":"2026-07-27T18:08:31Z","stars_delta_30d":50,"open_issues_delta_30d":8},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:39:03.219Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-24T00:02:13.384Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-24T00:02:13.384Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-24T00:02:13.384Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["You're working with NVIDIA NeMo models and require seamless integration.","Your project demands semantic-based deduplication techniques."],"when_not_to_use":["Your dataset doesn't align with NVIDIA hardware specifications.","You prefer data curation tools that do not emphasize semantic processing."],"source":"enrich:decision_facts","observed_at":"2026-07-12T17:35:46.481Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Scalable toolkit for data pre-processing tailored to LLMs, featuring deduplication and quality checks."}]}}