DS-1000

Enrichment pending

[ICML 2023] Data and code release for the paper "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation".

GitHub Website

GraphCanon updated today · GitHub synced today

273

Stars

Forks

Open issues

Watchers

Last push

Python CC-BY-SA-4.0Created Nov 15, 2022

Trust & integrity

Full report

Maintenance: Dormant (619d since push); As of today · Source: github_public_v1
Provenance: Not a fork · Organization account; As of today · Source: github_public_v1
Security (OSV): No lockfile; As of today · Source: none

Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.

Overview

[ICML 2023] Data and code release for the paper "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation".

Capability facts

Languages: python
Source: github.language · Jul 11, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Jul 11, 2026)

python test_ds1000.py

Source link

README

the test code also needs: pip install datasets tqdm

python test_ds1000.py


Expected output:

``` Codex002
           count   mean
lib                    
Matplotlib   155  0.548
Numpy        220  0.432
Pandas       291  0.265
Pytorch       68  0.397
Scipy        106  0.349
Sklearn      115  0.435
Tensorflow    45  0.378

DS-1000 overall
mean      0.388

See also results on new models in the results folder.

The test script executes generated code, so your own sandbox is strongly encouraged, but the reference code and provided solutions seems safe to run. Your sandbox has to allow some file operations (e.g. saving plot in matplotlib) and os operations (e.g. tensorflow, sklearn)