--- title: "tree-of-thought-llm" type: "tool" slug: "princeton-nlp-tree-of-thought-llm" canonical_url: "https://www.graphcanon.com/tools/princeton-nlp-tree-of-thought-llm" github_url: "https://github.com/princeton-nlp/tree-of-thought-llm" homepage_url: "https://arxiv.org/abs/2305.10601" stars: 6025 forks: 620 primary_language: "Python" license: "MIT" categories: ["llm-frameworks", "evaluation-observability"] tags: ["tree-search", "llm", "python", "large-language-models", "prompting", "tree-of-thoughts"] updated_at: "2026-07-07T17:34:44.787114+00:00" --- # tree-of-thought-llm > [NeurIPS 2023] Tree of Thoughts: Deliberate Problem Solving with Large Language Models [NeurIPS 2023] Tree of Thoughts: Deliberate Problem Solving with Large Language Models ## Facts - Repository: https://github.com/princeton-nlp/tree-of-thought-llm - Homepage: https://arxiv.org/abs/2305.10601 - Stars: 6,025 · Forks: 620 · Open issues: 8 · Watchers: 119 - Primary language: Python - License: MIT - Last pushed: 2025-01-16T20:02:00+00:00 ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) - [Evaluation & Observability](/categories/evaluation-observability.md) ## Tags tree-search, llm, python, large-language-models, prompting, tree-of-thoughts ## Related tools - [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,659) - [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019) - [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347) - [langflow](/tools/langflow-ai-langflow.md) - Langflow is a powerful platform for building and deploying AI-powered agents and workflows. (★ 151,298) - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575) - [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702) - [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,711) - [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,610) ## README (excerpt) ```text # Official Repo of Tree of Thoughts (ToT)

Build License

Official implementation for paper [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/abs/2305.10601) with code, prompts, model outputs. Also check [its tweet thread](https://twitter.com/ShunyuYao12/status/1659357547474681857) in 1min. ## Setup 1. Set up OpenAI API key and store in environment variable ``OPENAI_API_KEY`` (see [here](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety)). 2. Install `tot` package in two ways: - Option 1: Install from PyPI ```bash pip install tree-of-thoughts-llm ``` - Option 2: Install from source ```bash git clone https://github.com/princeton-nlp/tree-of-thought-llm cd tree-of-thought-llm pip install -r requirements.txt pip install -e . # install `tot` package ``` ## Quick Start The following minimal script will attempt to solve the game of 24 with `4 5 6 10` (might be a bit slow as it's using GPT-4): ```python import argparse from tot.methods.bfs import solve from tot.tasks.game24 import Game24Task args = argparse.Namespace(backend='gpt-4', temperature=0.7, task='game24', naive_run=False, prompt_sample=None, method_generate='propose', method_evaluate='value', method_select='greedy', n_generate_sample=1, n_evaluate_sample=3, n_select_sample=5) task = Game24Task() ys, infos = solve(args, task, 900) print(ys[0]) ``` And the output would be something like (note it's not deterministic, and sometimes the output can be wrong): ``` 10 - 4 = 6 (left: 5 6 6) 5 * 6 = 30 (left: 6 30) 30 - 6 = 24 (left: 24) Answer: (5 * (10 - 4)) - 6 = 24 ``` ## Paper Experiments Run experiments via ``sh scripts/{game24, text, crosswords}/{standard_sampling, cot_sampling, bfs}.sh``, except in crosswords we use a DFS algorithm for ToT, which can be run via ``scripts/crosswords/search_crosswords-dfs.ipynb``. The very simple ``run.py`` implements the ToT + BFS algorithm, as well as the naive IO/CoT sampling. Some key arguments: - ``--naive_run``: if True, run naive IO/CoT sampling instead of ToT + BFS. - ``--prompt_sample`` (choices=[``standard``, ``cot``]): sampling prompt - ``--method_generate`` (choices=[``sample``, ``propose``]): thought generator, whether to sample independent thoughts (used in Creative Writing) or propose sequential thoughts (used in Game of 24) - ``--method_evaluate`` (choices=[``value``, ``vote``]): state evaluator, whether to use the value states independently (used in Game of 24) or vote on states together (used in Creative Writing) - ``--n_generate_sample``: number of times to prompt for thought generation - ``--n_evaluate_sample``: number of times to prompt for state evaluation - ``--n_select_sample``: number of states to keep from each step (i.e. ``b`` in the paper's ToT + BFS algorithm) ## Paper Trajectories ``logs/`` contains all the trajectories from the paper's experiments, except for ``logs/game24/gpt-4_0.7_propose1_value3_greedy5_start900_end1000.json`` which was reproduced after the paper (as the original experiment was done in a notebook) and achieved a 69\% score instead of the original 74\% score due to randomness in GPT decoding. We hope to aggregate multiple runs in the future to account for sampling randomness and update the paper, but this shouldn't affect the main conclusions of the paper. ## How to Add A New Task Setting up a new task is easy, and mainly involves two steps. * Set up a new task c ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/princeton-nlp-tree-of-thought-llm`](/api/graphcanon/tools/princeton-nlp-tree-of-thought-llm) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_