{"data":{"slug":"xlang-ai-ds-1000","name":"DS-1000","tagline":"Benchmark and code for evaluating large language models on data science tasks","github_url":"https://github.com/xlang-ai/DS-1000","owner":"xlang-ai","repo":"DS-1000","owner_avatar_url":"https://avatars.githubusercontent.com/u/128829376?v=4","primary_language":"Python","stars":276,"forks":31,"topics":["benchmark","code-generation","data-science","large-language-models","semantic-parsing"],"archived":false,"github_pushed_at":"2024-10-30T17:43:46+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/xlang-ai-ds-1000","markdown_url":"https://www.graphcanon.com/tools/xlang-ai-ds-1000.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xlang-ai-ds-1000","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xlang-ai-ds-1000","description":"[ICML 2023] Data and code release for the paper \"DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation\".","homepage_url":"https://ds1000-code-gen.github.io","license":"CC-BY-SA-4.0","open_issues":2,"watchers":7,"ai_summary":"Provides benchmark and code generation assessment tools for large-language-models focused on data science applications. Involves testing the reliability and accuracy of generated code across various libraries including Matplotlib, Numpy, Pandas, Pytorch, Scipy, Sklearn, Tensorflow.","readme_excerpt":"# the test code also needs: pip install datasets tqdm\npython test_ds1000.py\n```\n\nExpected output:\n\n``` Codex002\n           count   mean\nlib                    \nMatplotlib   155  0.548\nNumpy        220  0.432\nPandas       291  0.265\nPytorch       68  0.397\nScipy        106  0.349\nSklearn      115  0.435\nTensorflow    45  0.378\n\nDS-1000 overall\nmean      0.388\n```\n\nSee also results on new models in the [results](./results) folder.\n\nThe test script executes generated code, so your own sandbox is strongly encouraged, but the reference code and provided solutions seems safe to run.\nYour sandbox has to allow some file operations (e.g. saving plot in matplotlib) and os operations (e.g. tensorflow, sklearn)","github_created_at":"2022-11-15T11:49:27+00:00","created_at":"2026-07-11T23:46:29.06028+00:00","updated_at":"2026-08-05T18:01:23.035461+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"benchmark","name":"benchmark"},{"slug":"code-generation","name":"code generation"},{"slug":"data-science","name":"data-science"},{"slug":"large-language-models","name":"large language models"},{"slug":"semantic-parsing","name":"semantic-parsing"}],"trust":{"provenance":{"is_fork":false,"github_id":566303764,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-05T18:01:22.227Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":644,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:46:30.654Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-05T18:01:22.707Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-05T18:01:22.707Z"},"license_spdx":{"value":"CC-BY-SA-4.0","source":"github.license","observed_at":"2026-08-05T18:01:22.707Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you want to assess how well a large language model can generate reliable and accurate code for data science projects involving popular Python libraries."],"when_not_to_use":["Avoid using DS-1000 if your project does not involve data science or if the models do not generate code in Python.","It is unsuitable for evaluating text generation abilities unrelated to coding, such as natural language processing tasks."],"source":"enrich:decision_facts","observed_at":"2026-07-17T05:15:28.246Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"The DS-1000 benchmark evaluates the code generation capabilities of large language models for data science tasks across Python libraries like Matplotlib, Numpy, Pandas, etc."}]}}