{"data":{"slug":"rlancemartin-auto-evaluator","name":"auto-evaluator","tagline":"A lightweight evaluation tool for question-answering using Langchain","github_url":"https://github.com/rlancemartin/auto-evaluator","owner":"rlancemartin","repo":"auto-evaluator","owner_avatar_url":"https://avatars.githubusercontent.com/u/122662504?v=4","primary_language":"Python","stars":1105,"forks":92,"topics":[],"archived":false,"github_pushed_at":"2023-05-10T02:00:18+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/rlancemartin-auto-evaluator","markdown_url":"https://www.graphcanon.com/tools/rlancemartin-auto-evaluator.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/rlancemartin-auto-evaluator","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=rlancemartin-auto-evaluator","description":"Evaluation tool for LLM QA chains","homepage_url":"https://autoevaluator.langchain.com/","license":null,"open_issues":3,"watchers":7,"ai_summary":"Auto-evaluator is an evaluation tool for LLM QA chains that allows users to input documents and generate question-answer pairs with LLMs such as GPT-3.5-turbo, evaluate the generated answers, and explore scoring across various chain configurations.","readme_excerpt":"# `Auto-evaluator` :brain: :memo:\n\n> **Note**\n> See the HuggingFace space for this app: https://huggingface.co/spaces/rlancemartin/auto-evaluator\n\n> **Note**\n> See the hosted app: https://autoevaluator.langchain.com/\n\n> **Note**\n> Code for the hosted app is also open source: https://github.com/langchain-ai/auto-evaluator\n\nThis is a lightweight evaluation tool for question-answering using Langchain to:\n\n- Ask the user to input a set of documents of interest\n\n- Apply an LLM (`GPT-3.5-turbo`) to auto-generate `question`-`answer` pairs from these docs\n\n- Generate a question-answering chain with a specified set of UI-chosen configurations\n\n- Use the chain to generate a response to each `question`\n\n- Use an LLM (`GPT-3.5-turbo`) to score the response relative to the `answer`\n\n- Explore scoring across various chain configurations\n\n**Run as Streamlit app**\n\n`pip install -r requirements.txt`\n\n`streamlit run auto-evaluator.py`\n\n**Inputs**\n\n`num_eval_questions` - Number of questions to auto-generate (if the user does not supply an eval set)\n\n`split_method` - Method for text splitting\n\n`chunk_chars` - Chunk size for text splitting\n \n`overlap` - Chunk overlap for text splitting\n  \n`embeddings` - Embedding method for chunks\n \n`retriever_type` - Chunk retrieval method\n\n`num_neighbors` - Neighbors for retrieval \n\n`model` - LLM for summarization of retrieved chunks \n\n`grade_prompt` - Prompt choice for model self-grading\n\n**Blog**\n\nhttps://blog.langchain.dev/auto-eval-of-question-answering-tasks/\n\n**UI**\n\n\n\n**Disclaimer**\n\n```You will need an OpenAI API key with access to `GPT-4` and an Anthropic API key to take advantage of all of the default dashboard model settings. However, additional models (e.g., from Hugging Face) can be easily added to the app.```","github_created_at":"2023-04-14T23:46:38+00:00","created_at":"2026-07-11T10:44:19.723693+00:00","updated_at":"2026-08-08T06:03:35.203462+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"evaluation","name":"evaluation"},{"slug":"gpt-3-5-turbo","name":"gpt-3.5-turbo"},{"slug":"langchain","name":"langchain"},{"slug":"llm","name":"llm"},{"slug":"question-answering","name":"question-answering"}],"trust":{"provenance":{"is_fork":false,"github_id":628105790,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-08T06:03:34.460Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1186,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":118,"high_count":0,"last_scan_at":"2026-07-11T10:44:21.270Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-08T06:03:34.918Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-08T06:03:34.918Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use when you need a lightweight solution for testing question-answering capabilities of Langchain models.","If you are working with GPT-3.5-turbo or other LLMs that integrate with Langchain, and require a streamlined way to auto-generate evaluations.","Suitable if your project involves the exploration of scoring across various chain configurations for QA tasks."],"when_not_to_use":["Avoid using this tool when you do not have access to an OpenAI API key providing access to GPT-4, as it uses that by default for optimal settings.","If you are looking for a tool that does not require you to input documents for question generation and prefer a more customized prompt approach rather than the auto-generation feature."],"source":"enrich:decision_facts","observed_at":"2026-07-15T08:22:17.931Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Auto-evaluator is a Python-based tool designed for evaluating LLM QA chains with the capability to auto-generate question-answer pairs from user-provided documents and evaluate answers using configurations chosen via UI."}]}}