{"data":{"slug":"rlancemartin-auto-evaluator","name":"auto-evaluator","tagline":"Evaluation tool for LLM QA chains","github_url":"https://github.com/rlancemartin/auto-evaluator","owner":"rlancemartin","repo":"auto-evaluator","owner_avatar_url":"https://avatars.githubusercontent.com/u/122662504?v=4","primary_language":"Python","stars":1102,"forks":92,"topics":[],"archived":false,"github_pushed_at":"2023-05-10T02:00:18+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/rlancemartin-auto-evaluator","markdown_url":"https://www.graphcanon.com/tools/rlancemartin-auto-evaluator.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/rlancemartin-auto-evaluator","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=rlancemartin-auto-evaluator","description":"Evaluation tool for LLM QA chains","homepage_url":"https://autoevaluator.langchain.com/","license":null,"open_issues":3,"watchers":7,"ai_summary":null,"readme_excerpt":"# `Auto-evaluator` :brain: :memo:\n\n> **Note**\n> See the HuggingFace space for this app: https://huggingface.co/spaces/rlancemartin/auto-evaluator\n\n> **Note**\n> See the hosted app: https://autoevaluator.langchain.com/\n\n> **Note**\n> Code for the hosted app is also open source: https://github.com/langchain-ai/auto-evaluator\n\nThis is a lightweight evaluation tool for question-answering using Langchain to:\n\n- Ask the user to input a set of documents of interest\n\n- Apply an LLM (`GPT-3.5-turbo`) to auto-generate `question`-`answer` pairs from these docs\n\n- Generate a question-answering chain with a specified set of UI-chosen configurations\n\n- Use the chain to generate a response to each `question`\n\n- Use an LLM (`GPT-3.5-turbo`) to score the response relative to the `answer`\n\n- Explore scoring across various chain configurations\n\n**Run as Streamlit app**\n\n`pip install -r requirements.txt`\n\n`streamlit run auto-evaluator.py`\n\n**Inputs**\n\n`num_eval_questions` - Number of questions to auto-generate (if the user does not supply an eval set)\n\n`split_method` - Method for text splitting\n\n`chunk_chars` - Chunk size for text splitting\n \n`overlap` - Chunk overlap for text splitting\n  \n`embeddings` - Embedding method for chunks\n \n`retriever_type` - Chunk retrieval method\n\n`num_neighbors` - Neighbors for retrieval \n\n`model` - LLM for summarization of retrieved chunks \n\n`grade_prompt` - Prompt choice for model self-grading\n\n**Blog**\n\nhttps://blog.langchain.dev/auto-eval-of-question-answering-tasks/\n\n**UI**\n\n\n\n**Disclaimer**\n\n```You will need an OpenAI API key with access to `GPT-4` and an Anthropic API key to take advantage of all of the default dashboard model settings. However, additional models (e.g., from Hugging Face) can be easily added to the app.```","github_created_at":"2023-04-14T23:46:38+00:00","created_at":"2026-07-11T10:44:19.723693+00:00","updated_at":"2026-07-11T10:44:23.538031+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":628105790,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:44:20.408Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1158,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":118,"high_count":0,"last_scan_at":"2026-07-11T10:44:21.270Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:44:20.840Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T10:44:20.840Z"}}}}