DS-1000
Benchmark and code for evaluating large language models on data science tasks
GraphCanon updated 2w · GitHub synced 2w
Decision brief
The DS-1000 benchmark evaluates the code generation capabilities of large language models for data science tasks across Python libraries like Matplotlib, Numpy, Pandas, etc.
Good fit when
- When you want to assess how well a large language model can generate reliable and accurate code for data science projects involving popular Python libraries.
Avoid when
- Avoid using DS-1000 if your project does not involve data science or if the models do not generate code in Python.
- It is unsuitable for evaluating text generation abilities unrelated to coding, such as natural language processing tasks.
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (644d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install DS-1000Similar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Provides benchmark and code generation assessment tools for large-language-models focused on data science applications. Involves testing the reliability and accuracy of generated code across various libraries including Matplotlib, Numpy, Pandas, Pytorch, Scipy, Sklearn, Tensorflow.
Capability facts
- Languages
- python
Source: github.language · Aug 5, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Tags
README
the test code also needs: pip install datasets tqdm
python test_ds1000.py
Expected output:
``` Codex002
count mean
lib
Matplotlib 155 0.548
Numpy 220 0.432
Pandas 291 0.265
Pytorch 68 0.397
Scipy 106 0.349
Sklearn 115 0.435
Tensorflow 45 0.378
DS-1000 overall
mean 0.388
See also results on new models in the results folder.
The test script executes generated code, so your own sandbox is strongly encouraged, but the reference code and provided solutions seems safe to run. Your sandbox has to allow some file operations (e.g. saving plot in matplotlib) and os operations (e.g. tensorflow, sklearn)
For agents
This page has a .md twin and JSON over the API.