chain-of-thought-hub
Benchmarking large language models' complex reasoning ability with chain-of-thought prompting
GraphCanon updated 2w · GitHub synced 2w
Decision brief
Chain-of-Thought Hub measures the performance of large language models (LLMs) on complex tasks by using carefully selected datasets across various domains such as math, science, coding, and knowledge. It evaluates if LLM
Good fit when
- Use Chain-of-Thought Hub when you need to benchmark smaller LLMs against larger ones for complex reasoning abilities.
- Consider using this tool if your project aims to assess the performance of models in detailed tasks that require nuanced understanding and multi-step reasoning.
Avoid when
- Do not use Chain-of-Thought Hub if your focus is on general conversational capabilities rather than specific, challenging problem-solving tasks.
- Avoid this tool if you are primarily interested in simpler language processing tasks that do not involve chain-of-thought prompting or complex datasets.
- Requirements:
- Min 8 GB RAM; Chain-of-Thought Hub is designed to be integrated into environments for evaluating LLMs using Jupyter Notebooks
Observed Jul 11, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (732d since push)
- As of 2w
- Provenance
- Not a fork · Personal account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/FranxYao/chain-of-thought-hubSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Chain-of-Thought Hub measures the performance of Large Language Models on complex tasks including math, science, symbolic manipulation, coding, and long-context understanding. It aims to assess if smaller models can match or compete with larger ones in these domains.
Capability facts
- Languages
- jupyter notebook
Source: github.language · Aug 6, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 6, 2026)
| Main | HumanEval | Python coding |Source link
Tags
README
Chain-of-Thought Hub: Measuring LLMs' Reasoning Performance
"A fantasy graph illustrating a chain of stars in a dark night with blue sky, digital art, super resolution". Midjourney V5
By Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot, Wenhu Chen
From University of Edinburgh, University of Washington, Allen Institute for AI, University of Waterloo
Recently, there are a lot of progress in LLMs. Many claim that a small model less than 10B can achieve comparable performance to GPT-3.5. Really?
In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle. The difference comes out when *the complexity of the task reaches a sufficient threshold* — GPT-4 is more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5. -- GPT-4 release blog
The key differentiator is whether a model can do complex tasks, like the old saying: "chit-chat is cheap, show me the reasoning." This is why we compile a list of complex reasoning tasks including math (GSM8K), science (MATH, TheoremQA), symbolic (BBH), knowledge (MMLU, C-Eval), coding (HumanEval), factual (SummEdits), and long-context (RepoBench, Qspr, QALT, BkSS) to measure the models' performance on challenging tasks.
More importantly, we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications. When this comes, chain-of-thought prompt engineering will be the next-generation system calls and shell scripts.
The credibility of chain-of-thought hub comes from the very carefully mediculously picked datasets and models that can clearly help the development of LLMs. The resutls and scripts from Chain-of-thought Hub is being used and referred by leading industrial and academic organizations in the space of large language models. We devide the tasks into three categories: main, experimental, and long-context.
- Main: datasets that are stable and consistently referred by places where LLMs are built.
- Experimental: datasets that has the potential to test future LLM capabilities.
- Long-context: datasets that require reasoning over very long context, an important direction of future LLMs.
[List of datasets we consider]
| Section | Dataset | Description |
|---|---|---|
| Main | GSM8K | Grade-level math word problems |
| Main | MATH | Competition-level math and science problems |
| Main | MMLU | Multi-discipline knowledge |
| Main | BBH | Challenging language and symbolic reasoning |
| Main | HumanEval | Python coding |
| Main | C-Eval | Chineses multi-discipline knowledge |
| Experimental | TheoremQA | Theorem proving |
| Experimental | SummEdits | Factual reasoning |
| Long Ctx | Qspr | Question answering over research papers |
| Long Ctx | QALT | Multiple-choice questions over long articles and stories |
| Long Ctx | BkSS | Reordering of summaries of parts of novels |
[Call for contribution]: would love to invite community members to:
- Send a PR to fill in a missing number in the table
- Raise an issue to suggest / brainstorm a new task / benchmark that measures reasoning over very long context
- Raise an issue to suggest / brainstorm a new task / benchmark that measures complex API calls and tool usage
- Raise an issue to suggest other good tasks / benchmarks that can clea
For agents
This page has a .md twin and JSON over the API.