---
title: "DS-1000"
type: "tool"
slug: "xlang-ai-ds-1000"
canonical_url: "https://www.graphcanon.com/tools/xlang-ai-ds-1000"
github_url: "https://github.com/xlang-ai/DS-1000"
homepage_url: "https://ds1000-code-gen.github.io"
stars: 273
forks: 31
primary_language: "Python"
license: "CC-BY-SA-4.0"
archived: false
categories: ["llm-frameworks", "model-training", "evaluation-observability"]
tags: ["data-science", "benchmark", "python", "large-language-models", "semantic-parsing", "code-generation"]
updated_at: "2026-07-11T23:46:41.952761+00:00"
---

# DS-1000

> [ICML 2023] Data and code release for the paper "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation".

[ICML 2023] Data and code release for the paper "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation".

## Facts

- Repository: https://github.com/xlang-ai/DS-1000
- Homepage: https://ds1000-code-gen.github.io
- Stars: 273 · Forks: 31 · Open issues: 2 · Watchers: 7
- Primary language: Python
- License: CC-BY-SA-4.0
- Last pushed: 2024-10-30T17:43:46+00:00

## Trust & health

_Signals computed from public GitHub metadata. Not a security guarantee._

- Maintenance: Dormant (computed 2026-07-11T23:46:30.177Z)
- Security scan: No lockfile (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-11T23:46:30.654Z
- Full report: [trust report](/tools/xlang-ai-ds-1000/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/xlang-ai-ds-1000/trust)

## Categories

- [LLM Frameworks](/categories/llm-frameworks.md)
- [Model Training](/categories/model-training.md)
- [Evaluation & Observability](/categories/evaluation-observability.md)

## Tags

data-science, benchmark, python, large-language-models, semantic-parsing, code-generation

## Category neighbours (exploratory)

_Same-category tools for discovery only - not curated alternatives. Cap shown at six._

- [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active]
- [tensorflow](/tools/tensorflow-tensorflow.md) - An Open Source Machine Learning Framework for Everyone (★ 196,300) [Very active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active]
- [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 162,482) [Very active]

_+ 2 more not listed._

## README (excerpt)

_Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._

````text
# the test code also needs: pip install datasets tqdm
python test_ds1000.py
```

Expected output:

``` Codex002
           count   mean
lib                    
Matplotlib   155  0.548
Numpy        220  0.432
Pandas       291  0.265
Pytorch       68  0.397
Scipy        106  0.349
Sklearn      115  0.435
Tensorflow    45  0.378

DS-1000 overall
mean      0.388
```

See also results on new models in the [results](./results) folder.

The test script executes generated code, so your own sandbox is strongly encouraged, but the reference code and provided solutions seems safe to run.
Your sandbox has to allow some file operations (e.g. saving plot in matplotlib) and os operations (e.g. tensorflow, sklearn)
````

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/xlang-ai-ds-1000`](/api/graphcanon/tools/xlang-ai-ds-1000)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_