GSM-IC
Arithmetic reasoning dataset with irrelevant context to test model distractibility
GraphCanon updated 3w · GitHub synced 3w
Decision brief
A benchmark dataset for assessing the capability of language models in solving arithmetic problems amidst distractions.
Good fit when
- When evaluating how well a model can ignore irrelevant context to solve grade-school math problems from GSM8K.
- For researchers aiming to measure and improve the distraction resistance of AI models on specific task performance.
Avoid when
- If your aim is to evaluate arithmetic reasoning capabilities without any distractions or unnecessary textual information added to the problems.
- When you require a dataset for purely numerical analysis tasks that do not involve understanding language text or filtering out irrelevant information.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Archived (1264d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/google-research-datasets/GSM-ICSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A benchmark dataset derived from GSM8K, appending irrelevant statements to grade-school math problems. Evaluates a language model's capability in ignoring distractions to solve arithmetic problems.
Capability facts
No sourced capability facts yet. Facts appear after ingest scans repo manifests (Dockerfile, package.json, MCP configs).
Categories
Tags
README
Grade-School Math with Irrelevant Context (GSM-IC)
This repository contains the dataset Grade-School Math with Irrelevant Context (GSM-IC) used in this paper: Large Language Models Can Be Easily Distracted by Irrelevant Context.
Data Format
GSM8K_validation.jsonl: the development split of GSM8K dataset used in the experiments.
| Field name | Value |
|---|---|
| question | Input question. |
| answer | The ground truth answer. |
| n_steps | The number of intermediate steps to calculate the answer. |
GSM-IC_2step.json: GSM-IC split with problems that require 2 intermediate steps.
| Field name | Value |
|---|---|
| original_question | Original question from the GSM8K development set. |
| new_question | The new question with irrelevant context added to the original question. |
| answer | The ground truth answer. |
| n_steps | The number of intermediate steps to calculate the answer. |
| role_label, number_label, sentence_label | Categories of the added irrelevant context. Needed for result analysis, not needed for model prediction. |
| role, number, sentence_template | Added irrelevant context. Not needed for experiments. |
GSM-IC_mstep.json: GSM-IC split with problems that require more than 2 intermediate steps. Same format asGSM-IC_2step.json.
Citation
If you use the data released through this repository, please cite the following paper:
@article{shi2023large,
title={Large Language Models Can Be Easily Distracted by Irrelevant Context},
author={Shi, Freda and Chen, Xinyun and Misra, Kanishka and Scales, Nathan and Dohan, David and Chi, Ed and Schärli, Nathanael and Zhou, Denny},
journal={arXiv preprint arXiv:2302.00093},
year={2023}
}
For agents
This page has a .md twin and JSON over the API.