auto-evaluator
A lightweight evaluation tool for question-answering using Langchain
GraphCanon updated 2w · GitHub synced 2w
Decision brief
Auto-evaluator is a Python-based tool designed for evaluating LLM QA chains with the capability to auto-generate question-answer pairs from user-provided documents and evaluate answers using configurations chosen via UI.
Good fit when
- Use when you need a lightweight solution for testing question-answering capabilities of Langchain models.
- If you are working with GPT-3.5-turbo or other LLMs that integrate with Langchain, and require a streamlined way to auto-generate evaluations.
Avoid when
- Avoid using this tool when you do not have access to an OpenAI API key providing access to GPT-4, as it uses that by default for optimal settings.
- If you are looking for a tool that does not require you to input documents for question generation and prefer a more customized prompt approach rather than the auto-generation feature.
Observed Jul 15, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (1186d since push)
- As of 2w
- Provenance
- Not a fork · Personal account
- As of 2w
- Security (OSV)
- 118 low (118 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install auto-evaluatorSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Auto-evaluator is an evaluation tool for LLM QA chains that allows users to input documents and generate question-answer pairs with LLMs such as GPT-3.5-turbo, evaluate the generated answers, and explore scoring across various chain configurations.
Capability facts
- Languages
- python
Source: github.language · Aug 8, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 8, 2026)
```You will need an OpenAI API key with access to `GPT-4` and an Anthropic API key to take advantage of all of the default dashboard model settings. However,Source link
Source: README excerpt (regex_v1, Aug 8, 2026)
> See the hosted app: https://autoevaluator.langchain.com/Source link
Source: README excerpt (regex_v1, Aug 8, 2026)
```You will need an OpenAI API key with access to `GPT-4` and an Anthropic API key to take advantage of all ofSource link
Source: README excerpt (regex_v1, Aug 8, 2026)
`pip install -r requirements.txt`Source link
Tags
README
Auto-evaluator :brain: :memo:
Note See the HuggingFace space for this app: https://huggingface.co/spaces/rlancemartin/auto-evaluator
Note See the hosted app: https://autoevaluator.langchain.com/
Note Code for the hosted app is also open source: https://github.com/langchain-ai/auto-evaluator
This is a lightweight evaluation tool for question-answering using Langchain to:
-
Ask the user to input a set of documents of interest
-
Apply an LLM (
GPT-3.5-turbo) to auto-generatequestion-answerpairs from these docs -
Generate a question-answering chain with a specified set of UI-chosen configurations
-
Use the chain to generate a response to each
question -
Use an LLM (
GPT-3.5-turbo) to score the response relative to theanswer -
Explore scoring across various chain configurations
Run as Streamlit app
pip install -r requirements.txt
streamlit run auto-evaluator.py
Inputs
num_eval_questions - Number of questions to auto-generate (if the user does not supply an eval set)
split_method - Method for text splitting
chunk_chars - Chunk size for text splitting
overlap - Chunk overlap for text splitting
embeddings - Embedding method for chunks
retriever_type - Chunk retrieval method
num_neighbors - Neighbors for retrieval
model - LLM for summarization of retrieved chunks
grade_prompt - Prompt choice for model self-grading
Blog
https://blog.langchain.dev/auto-eval-of-question-answering-tasks/
UI
Disclaimer
You will need an OpenAI API key with access to `GPT-4` and an Anthropic API key to take advantage of all of the default dashboard model settings. However, additional models (e.g., from Hugging Face) can be easily added to the app.
For agents
This page has a .md twin and JSON over the API.