Chain-of-Thought Hub: Measuring LLMs' Reasoning Performance

"A fantasy graph illustrating a chain of stars in a dark night with blue sky, digital art, super resolution". Midjourney V5

---

By Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot, Wenhu Chen

From University of Edinburgh, University of Washington, Allen Institute for AI, University of Waterloo

[paper] [blog] [twitter]

Recently, there are a lot of progress in LLMs. Many claim that a small model less than 10B can achieve comparable performance to GPT-3.5. Really?

In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle. The difference comes out when *the complexity of the task reaches a sufficient threshold* — GPT-4 is more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5. -- GPT-4 release blog

The key differentiator is whether a model can do complex tasks, like the old saying: "chit-chat is cheap, show me the reasoning." This is why we compile a list of complex reasoning tasks including math (GSM8K), science (MATH, TheoremQA), symbolic (BBH), knowledge (MMLU, C-Eval), coding (HumanEval), factual (SummEdits), and long-context (RepoBench, Qspr, QALT, BkSS) to measure the models' performance on challenging tasks.

More importantly, we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications. When this comes, chain-of-thought prompt engineering will be the next-generation system calls and shell scripts.

The credibility of chain-of-thought hub comes from the very carefully mediculously picked datasets and models that can clearly help the development of LLMs. The resutls and scripts from Chain-of-thought Hub is being used and referred by leading industrial and academic organizations in the space of large language models. We devide the tasks into three categories: main, experimental, and long-context.

• Main: datasets that are stable and consistently referred by places where LLMs are built.
• Experimental: datasets that has the potential to test future LLM capabilities.
• Long-context: datasets that require reasoning over very long context, an important direction of future LLMs.

[List of datasets we consider]

Section	Dataset	Description
Main	GSM8K	Grade-level math word problems
Main	MATH	Competition-level math and science problems
Main	MMLU	Multi-discipline knowledge
Main	BBH	Challenging language and symbolic reasoning
Main	HumanEval	Python coding
Main	C-Eval	Chineses multi-discipline knowledge
Experimental	TheoremQA	Theorem proving
Experimental	SummEdits	Factual reasoning
Long Ctx	Qspr	Question answering over research papers
Long Ctx	QALT	Multiple-choice questions over long articles and stories
Long Ctx	BkSS	Reordering of summaries of parts of novels

[Call for contribution]: would love to invite community members to:

• Send a PR to fill in a missing number in the table
• Raise an issue to suggest / brainstorm a new task / benchmark that measures reasoning over very long context
• Raise an issue to suggest / brainstorm a new task / benchmark that measures complex API calls and tool usage
• Raise an issue to suggest other good tasks / benchmarks that can clea