{"data":{"slug":"franxyao-chain-of-thought-hub","name":"chain-of-thought-hub","tagline":"Benchmarking large language models' complex reasoning ability with chain-of-thought prompting","github_url":"https://github.com/FranxYao/chain-of-thought-hub","owner":"FranxYao","repo":"chain-of-thought-hub","owner_avatar_url":"https://avatars.githubusercontent.com/u/17723677?v=4","primary_language":"Jupyter Notebook","stars":2777,"forks":144,"topics":[],"archived":false,"github_pushed_at":"2024-08-04T09:40:18+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/franxyao-chain-of-thought-hub","markdown_url":"https://www.graphcanon.com/tools/franxyao-chain-of-thought-hub.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/franxyao-chain-of-thought-hub","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=franxyao-chain-of-thought-hub","description":"Benchmarking large language models' complex reasoning ability with chain-of-thought prompting","homepage_url":null,"license":"MIT","open_issues":27,"watchers":36,"ai_summary":"Chain-of-Thought Hub measures the performance of Large Language Models on complex tasks including math, science, symbolic manipulation, coding, and long-context understanding. It aims to assess if smaller models can match or compete with larger ones in these domains.","readme_excerpt":"# Chain-of-Thought Hub: Measuring LLMs' Reasoning Performance\n\n\n\"A fantasy graph illustrating a chain of stars in a dark night with blue sky, digital art, super resolution\". Midjourney V5\n\n----\n\n\nBy [Yao Fu](https://franxyao.github.io/), [Litu Ou](https://github.com/Leonard907), [Mingyu Chen](https://github.com/Spehhhhh), [Yuhao Wan](https://github.com/Yuhao-Wan), [Hao Peng](https://haopeng-nlp.github.io/), [Tushar Khot](https://allenai.org/team/tushark), [Wenhu Chen](https://wenhuchen.github.io/)\n\nFrom University of Edinburgh, University of Washington, Allen Institute for AI, University of Waterloo\n\n[[paper](https://arxiv.org/abs/2305.17306)] [[blog](https://yaofu.notion.site/Towards-Complex-Reasoning-the-Polaris-of-Large-Language-Models-c2b4a51355b44764975f88e6a42d4e75)]  [[twitter](https://twitter.com/Francis_YAO_/status/1663472109299937280)]\n\nRecently, there are a lot of progress in LLMs. Many claim that a small model less than 10B can achieve comparable performance to GPT-3.5. Really? \n\n> In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle. The difference comes out when **\\*the complexity of the task reaches a sufficient threshold\\*** — GPT-4 is more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5.  --  *GPT-4 release blog*\n\nThe key differentiator is whether a model can do **complex tasks**, like the old saying: \"chit-chat is cheap, show me the reasoning.\" This is why we compile a list of complex reasoning tasks including math (GSM8K), science (MATH, TheoremQA), symbolic (BBH), knowledge (MMLU, C-Eval), coding (HumanEval), factual (SummEdits), and long-context (RepoBench, Qspr, QALT, BkSS) to measure the models' performance on challenging tasks. \n\nMore importantly, we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications.\nWhen this comes, chain-of-thought prompt engineering will be the next-generation system calls and shell scripts. \n\nThe credibility of chain-of-thought hub comes from the very carefully mediculously picked datasets and models that can clearly help the development of LLMs. The resutls and scripts from Chain-of-thought Hub is being used and referred by leading industrial and academic organizations in the space of large language models. We devide the tasks into three categories: main, experimental, and long-context.\n* Main: datasets that are stable and consistently referred by places where LLMs are built.\n* Experimental: datasets that has the potential to test future LLM capabilities.\n* Long-context: datasets that require reasoning over very long context, an important direction of future LLMs.\n\n<details>\n  <summary>[List of datasets we consider]</summary>\n\n  | Section  | Dataset   | Description | \n  | -------  | -------   | ----------- |\n  | Main     | GSM8K     | Grade-level math word problems |\n  | Main     | MATH      | Competition-level math and science problems |\n  | Main     | MMLU      | Multi-discipline knowledge |\n  | Main     | BBH       | Challenging language and symbolic reasoning |\n  | Main     | HumanEval | Python coding |\n  | Main     | C-Eval    | Chineses multi-discipline knowledge |\n  | Experimental     | TheoremQA | Theorem proving |\n  | Experimental     | SummEdits | Factual reasoning |\n  | Long Ctx | Qspr      | Question answering over research papers |\n  | Long Ctx | QALT      | Multiple-choice questions over long articles and stories | \n  | Long Ctx | BkSS      | Reordering of summaries of parts of novels | \n</details>\n\n\n**[Call for contribution]**: would love to invite community members to:\n* Send a PR to fill in a missing number in the table\n* Raise an issue to suggest / brainstorm a new task / benchmark that measures **reasoning over very long context**\n* Raise an issue to suggest / brainstorm a new task / benchmark that measures **complex API calls and tool usage**\n* Raise an issue to suggest other good tasks / benchmarks that can clea","github_created_at":"2023-03-10T20:28:11+00:00","created_at":"2026-07-11T10:31:55.499269+00:00","updated_at":"2026-07-11T11:14:21.895764+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"complex-reasoning","name":"complex reasoning"},{"slug":"chain-of-thought-prompting","name":"chain-of-thought prompting"},{"slug":"llm-benchmarking","name":"llm-benchmarking"}],"trust":{"provenance":{"is_fork":false,"github_id":612382241,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:31:57.248Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":706,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:31:58.444Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T11:13:47.889Z"},"languages":{"value":["jupyter notebook"],"source":"github.language","observed_at":"2026-07-11T11:13:47.889Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T11:13:47.889Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Chain-of-Thought Hub is designed to be integrated into environments for evaluating LLMs using Jupyter Notebooks"],"min_ram_gb":8,"requires_docker":false},"constraints":{"min_ram_gb":8,"requires_docker":false},"when_to_use":["Use Chain-of-Thought Hub when you need to benchmark smaller LLMs against larger ones for complex reasoning abilities.","Consider using this tool if your project aims to assess the performance of models in detailed tasks that require nuanced understanding and multi-step reasoning."],"when_not_to_use":["Do not use Chain-of-Thought Hub if your focus is on general conversational capabilities rather than specific, challenging problem-solving tasks.","Avoid this tool if you are primarily interested in simpler language processing tasks that do not involve chain-of-thought prompting or complex datasets."],"source":"enrich:decision_facts","observed_at":"2026-07-11T11:14:21.575Z"},"constraint_facets":{"min_ram_gb":8,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Min 8 GB RAM; Chain-of-Thought Hub is designed to be integrated into environments for evaluating LLMs using Jupyter Notebooks"},{"label":"Adopt for","value":"Chain-of-Thought Hub measures the performance of large language models (LLMs) on complex tasks by using carefully selected datasets across various domains such as math, science, coding, and knowledge. It evaluates if LLM"},{"label":"License detail","value":"The MIT license permits the use of Chain-of-Thought Hub in both open source and commercial projects with acknowledgment."}]}}