--- title: "GSM-IC vs Auto-claude-code-research-in-sleep" type: "comparison" canonical_url: "https://www.graphcanon.com/compare/google-research-datasets-gsm-ic-vs-wanshuiyin-auto-claude-code-research-in-sleep" tools: ["google-research-datasets-gsm-ic", "wanshuiyin-auto-claude-code-research-in-sleep"] --- # GSM-IC vs Auto-claude-code-research-in-sleep *GraphCanon updated Sep 20, 2026* ## Verdict Pick GSM-IC if gSM-IC is a benchmark dataset that evaluates the distractibility of language models by introducing irrelevant context into grade-school math problems; pick Auto-claude-code-research-in-sleep if auto-claude-code-research-in-sleep provides specialized Markdown-based utilities for automating and enhancing autonomous ML research by connecting various models in an open framework. [GSM-IC](https://github.com/google-research-datasets/GSM-IC) reports 67 GitHub stars, 2 forks, and 1 open issues, last pushed Feb 13, 2023. [Auto-claude-code-research-in-sleep](https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep) has 16k stars, 1.4k forks, and 70 open issues, last pushed Sep 18, 2026. Figures are from public GitHub metadata via [GSM-IC's repository](https://github.com/google-research-datasets/GSM-IC) and [Auto-claude-code-research-in-sleep's repository](https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep). | | [GSM-IC](/tools/google-research-datasets-gsm-ic.md) | [Auto-claude-code-research-in-sleep](/tools/wanshuiyin-auto-claude-code-research-in-sleep.md) | | --- | --- | --- | | Tagline | GSM-IC benchmark evaluates distractibility of language models with grade-school math problems and irrelevant context. | Lightweight Markdown-only skills for autonomous ML research | | Stars | 67 | 16,383 | | Forks | 2 | 1,396 | | Open issues | 1 | 70 | | Language | - | Python | | Adopt for | GSM-IC is a benchmark dataset that evaluates the distractibility of language models by introducing irrelevant context into grade-school math problems. | Auto-claude-code-research-in-sleep provides specialized Markdown-based utilities for automating and enhancing autonomous ML research by connecting various models in an open framework. | | Persona | - | - | | Runtime | - | - | | License | License information is not provided in the repository. | MIT License, allowing for broad usage without restrictions on commercial use. | | Categories | Evaluation & Observability | AI Agents, Developer Tools, Evaluation & Observability | ## Trust and health _Sourced signals - not a safety guarantee. No winner column._ | | [GSM-IC](/tools/google-research-datasets-gsm-ic.md) | [Auto-claude-code-research-in-sleep](/tools/wanshuiyin-auto-claude-code-research-in-sleep.md) | | --- | --- | --- | | Maintenance | Archived (8%) | Very active (96%) | | Days since push | 1312d | 1d | | Archived on GitHub | Yes | No | | Open issues (now) | 1 | 70 | | Stars delta | 0 (30d) | +2.5k (30d) | | Open issues delta | 0 (30d) | +10 (30d) | | Owner type | Organization | User | | Full report | [trust report](/tools/google-research-datasets-gsm-ic/trust.md) | [trust report](/tools/wanshuiyin-auto-claude-code-research-in-sleep/trust.md) | ## Decision facts: GSM-IC - **Pricing:** unknown - Pricing details are not available. - **Requirements:** The dataset is derived from GSM8K and includes irrelevant context in problem descriptions to test the distractibility of language models. - **Adopt for:** GSM-IC is a benchmark dataset that evaluates the distractibility of language models by introducing irrelevant context into grade-school math problems. - **License detail:** License information is not provided in the repository. ## Decision facts: Auto-claude-code-research-in-sleep - **Pricing:** freemium - Free to use under MIT license with no explicit pricing model indicated, though users might incur costs based on the AI models and services they choose to integrate. - **Requirements:** Compatibility with diverse language model agents without requiring lock-in or specific frameworks; Utilizes Markdown for skills, aiming at a lightweight automation layer on top of ML research tasks - **Adopt for:** Auto-claude-code-research-in-sleep provides specialized Markdown-based utilities for automating and enhancing autonomous ML research by connecting various models in an open framework. - **License detail:** MIT License, allowing for broad usage without restrictions on commercial use. ## Choose when ### Choose GSM-IC if… - Pricing: Pricing details are not available.. - Requirements: The dataset is derived from GSM8K and includes irrelevant context in problem descriptions to test the distractibility of language models.. - Tags unique to GSM-IC: arithmetic reasoning, benchmark, distractibility, grade-school math. - Use GSM-IC when you need to assess how well a language model can focus on relevant information in arithmetic reasoning tasks, especially when faced with distracting irrelevant context. ### Choose Auto-claude-code-research-in-sleep if… - Pricing: Free to use under MIT license with no explicit pricing model indicated, though users might incur costs based on the AI models and services they choose to integrate.. - Requirements: Compatibility with diverse language model agents without requiring lock-in or specific frameworks; Utilizes Markdown for skills, aiming at a lightweight automation layer on top of ML research tasks. - Tags unique to Auto-claude-code-research-in-sleep: ai research, autonomous-agent, idea-generation, ml-research. - Also covers AI Agents, Developer Tools. - When you are looking to streamline idea discovery, experiment automation, and cross-model review loops specifically within the context of Python programming for machine learning research ## When NOT to use GSM-IC - Avoid GSM-IC if your evaluation focuses on the model's ability to solve math problems without any form of distraction or if the context is always assumed to be relevant. - Do not use GSM-IC if your primary interest is in evaluating the model's performance on pure arithmetic tasks without the added complexity of irrelevant context. ## When NOT to use Auto-claude-code-research-in-sleep - If you require a solution that is tightly integrated with a specific AI development platform or requires the use of proprietary models - When your research workflow demands real-time data analysis and visualization tools that Auto-claude-code-research-in-sleep does not directly support ## Common questions ### What is the difference between GSM-IC and Auto-claude-code-research-in-sleep? GSM-IC: GSM-IC benchmark evaluates distractibility of language models with grade-school math problems and irrelevant context.. Auto-claude-code-research-in-sleep: Lightweight Markdown-only skills for autonomous ML research. See the comparison table for live GitHub stats and shared categories. ### When should I choose GSM-IC over Auto-claude-code-research-in-sleep? Choose GSM-IC over Auto-claude-code-research-in-sleep when Pricing: Pricing details are not available.; Requirements: The dataset is derived from GSM8K and includes irrelevant context in problem descriptions to test the distractibility of language models.; Tags unique to GSM-IC: arithmetic reasoning, benchmark, distractibility, grade-school math; Use GSM-IC when you need to assess how well a language model can focus on relevant information in arithmetic reasoning tasks, especially when faced with distracting irrelevant context. ### When should I choose Auto-claude-code-research-in-sleep over GSM-IC? Choose Auto-claude-code-research-in-sleep over GSM-IC when Pricing: Free to use under MIT license with no explicit pricing model indicated, though users might incur costs based on the AI models and services they choose to integrate.; Requirements: Compatibility with diverse language model agents without requiring lock-in or specific frameworks; Utilizes Markdown for skills, aiming at a lightweight automation layer on top of ML research tasks; Tags unique to Auto-claude-code-research-in-sleep: ai research, autonomous-agent, idea-generation, ml-research; Also covers AI Agents, Developer Tools; When you are looking to streamline idea discovery, experiment automation, and cross-model review loops specifically within the context of Python programming for machine learning research. ### When should I avoid GSM-IC? Avoid GSM-IC if your evaluation focuses on the model's ability to solve math problems without any form of distraction or if the context is always assumed to be relevant. Do not use GSM-IC if your primary interest is in evaluating the model's performance on pure arithmetic tasks without the added complexity of irrelevant context. ### When should I avoid Auto-claude-code-research-in-sleep? If you require a solution that is tightly integrated with a specific AI development platform or requires the use of proprietary models When your research workflow demands real-time data analysis and visualization tools that Auto-claude-code-research-in-sleep does not directly support ### Is GSM-IC or Auto-claude-code-research-in-sleep more popular on GitHub? Auto-claude-code-research-in-sleep has more GitHub stars (16,383 vs 67). Stars measure visibility, not whether either tool fits your constraints. ### Are GSM-IC and Auto-claude-code-research-in-sleep open source? Yes - both are open-source projects on GitHub. ### Where can I find alternatives to GSM-IC or Auto-claude-code-research-in-sleep? GraphCanon lists graph-backed alternatives at [GSM-IC alternatives](/tools/google-research-datasets-gsm-ic/alternatives) and [Auto-claude-code-research-in-sleep alternatives](/tools/wanshuiyin-auto-claude-code-research-in-sleep/alternatives) ([GSM-IC markdown twin](/tools/google-research-datasets-gsm-ic/alternatives.md), [Auto-claude-code-research-in-sleep markdown twin](/tools/wanshuiyin-auto-claude-code-research-in-sleep/alternatives.md)), ranked by typed relationship edges rather than popularity votes. ### Is there a machine-readable version of this comparison? Yes. The markdown twin at [this comparison](/compare/google-research-datasets-gsm-ic-vs-wanshuiyin-auto-claude-code-research-in-sleep.md) mirrors this page for agents and LLM crawlers, with the same stats table and FAQ answers. ### Which is better maintained, GSM-IC or Auto-claude-code-research-in-sleep? GSM-IC: Archived. Auto-claude-code-research-in-sleep: Very active. Compare maintenance labels, days since push, and release cadence in the trust section below - stars alone do not measure maintenance. ### Where are the full trust reports for GSM-IC and Auto-claude-code-research-in-sleep? GraphCanon publishes per-repo trust reports with dated maintenance, provenance, and scan summaries: [GSM-IC trust report](/tools/google-research-datasets-gsm-ic/trust); [Auto-claude-code-research-in-sleep trust report](/tools/wanshuiyin-auto-claude-code-research-in-sleep/trust). --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/graph?tool=google-research-datasets-gsm-ic`](/api/graphcanon/graph?tool=google-research-datasets-gsm-ic) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_