auto-evaluator
A lightweight evaluation tool for question-answering using Langchain
GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026
94views this month
Decision brief
Auto-evaluator is a Python-based tool designed for evaluating LLM QA chains with the capability to auto-generate question-answer pairs from user-provided documents and evaluate answers using configurations chosen via UI.
Good fit when
- Use when you need a lightweight solution for testing question-answering capabilities of Langchain models.
- If you are working with GPT-3.5-turbo or other LLMs that integrate with Langchain, and require a streamlined way to auto-generate evaluations.
Avoid when
- Avoid using this tool when you do not have access to an OpenAI API key providing access to GPT-4, as it uses that by default for optimal settings.
- If you are looking for a tool that does not require you to input documents for question generation and prefer a more customized prompt approach rather than the auto-generation feature.
Observed Jul 15, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (1216d since push)
- As of Sep 8, 2026
- Provenance
- Not a fork · Personal account
- As of Sep 8, 2026
- Security (OSV)
- 118 low (118 low)
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install auto-evaluatorSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Auto-evaluator is an evaluation tool for LLM QA chains that allows users to input documents and generate question-answer pairs with LLMs such as GPT-3.5-turbo, evaluate the generated answers, and explore scoring across various chain configurations.
Capability facts
- Languages
- python
Source: github.language · Sep 8, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Sep 8, 2026)
```You will need an OpenAI API key with access to `GPT-4` and an Anthropic API key to take advantage of all of the default dashboard model settings. However,Source link
Source: README excerpt (regex_v1, Sep 8, 2026)
> See the hosted app: https://autoevaluator.langchain.com/Source link
Source: README excerpt (regex_v1, Sep 8, 2026)
```You will need an OpenAI API key with access to `GPT-4` and an Anthropic API key to take advantage of all ofSource link
Source: README excerpt (regex_v1, Sep 8, 2026)
`pip install -r requirements.txt`Source link
Tags
README
:brain: :memo: Note See the HuggingFace space for this app: https://huggingface.co/spaces/rlancemartin/auto evaluator Note See the hosted app: https://autoevaluator.langchain.com/ Note Code for the hosted app is also open source: https://github.com/langchain ai/auto evaluator Thi...
For agents
This page has a .md twin and JSON over the API.