⛓️🛠️ ChainForge

An open-source visual environment for battle-testing prompts to LLMs.

ChainForge is a data flow prompt engineering environment for analyzing and evaluating LLM responses. It enables rapid-fire, quick-and-dirty comparison of prompts, models, and response quality that goes beyond ad-hoc chatting with individual LLMs. With ChainForge, you can:

• Query multiple LLMs at once to test prompt ideas and variations quickly and effectively.
• Compare response quality across prompt permutations, across models, and across model settings to choose the best prompt and model for your use case.
• Setup evaluation metrics (scoring function) and immediately visualize results across prompts, prompt parameters, models, and model settings.
• Use AI to streamline this entire process: Create synthetic tables and input examples with built-in genAI features, or supercharge writing evals by prompting a model to give you starter code.

Read the docs to learn more. ChainForge comes with a number of example evaluation flows to give you a sense of what's possible, including 188 example flows generated from benchmarks in OpenAI evals.

ChainForge is built on ReactFlow and Flask.

For user-curated resources and learning materials, check out the 🌟Awesome ChainForge repo!

Table of Contents

• 👉 Documentation 📖
• Installation
• Example Experiments
• Share with Others
• Features (see the docs for more comprehensive info)
• Development and How to Cite

Installation

You can install ChainForge locally, or try it out on the web at https://chainforge.ai/play/. The web version of ChainForge has a limited feature set. In a locally installed version you can load API keys automatically from environment variables, write Python code to evaluate LLM responses, or query locally-run models hosted via Ollama.

To install Chainforge on your machine, make sure you have Python 3.8 or higher, then run

pip install chainforge

Once installed, do

chainforge serve

Open localhost:8000 in a Google Chrome, Firefox, Microsoft Edge, or Brave browser.

You can set your API keys by clicking the Settings icon in the top-right corner. If you prefer to not worry about this everytime you open ChainForge, we highly recommend that save your OpenAI, Anthropic, Google, etc API keys and/or Amazon AWS credentials to your local environment. For more details, see the How to Install.

Run using Docker

You can use our Dockerfile to run ChainForge locally using Docker Desktop:

• Build the Dockerfile:

  docker build -t chainforge .
  

• Run the image:

  docker run -p 8000:8000 chainforge
  

Now you can open the browser of your choice and open http://127.0.0.1:8000.

Supported providers

• OpenAI
• Anthropic
• Google Gemini
• DeepSeek
• HuggingFace (Inference and Endpoints)
• Together.ai
• Ollama API (locally-hosted models)
• Microsoft Azure OpenAI Endpoints
• Aleph Alpha
• Amazon Bedrock-hosted on-demand inference, including Anthropic Claude 3
• ...and any other provider through custom provider scripts!

Example experiments

We've prepared many example flows to give you a sense of what's possible with Chainforge. Click the "Example Flows" button on the top-right corner and select one. Here is a basic comparison