{"data":{"slug":"rungalileo-hallucination-index","name":"hallucination-index","tagline":"Initiative to evaluate and rank popular LLMs based on hallucination propensity","github_url":"https://github.com/rungalileo/hallucination-index","owner":"rungalileo","repo":"hallucination-index","owner_avatar_url":"https://avatars.githubusercontent.com/u/81123343?v=4","primary_language":null,"stars":116,"forks":8,"topics":["hallucinations","large-language-models","llm","llm-evaluation","openai","rag","retrieval-augmented-generation"],"archived":false,"github_pushed_at":"2025-07-28T18:28:09+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/rungalileo-hallucination-index","markdown_url":"https://www.graphcanon.com/tools/rungalileo-hallucination-index.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/rungalileo-hallucination-index","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=rungalileo-hallucination-index","description":"Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate. ","homepage_url":"https://www.rungalileo.io/hallucinationindex","license":null,"open_issues":1,"watchers":5,"ai_summary":"The repository evaluates the tendency of large language models (LLMs) to generate factual errors, known as 'hallucinations,' by testing model attributes such as context length and source openness. It includes a comprehensive assessment of 22 open-source and closed-source models using various prompting techniques across short, medium, and long contexts.","readme_excerpt":"# ๐ŸŒŸ LLM Hallucination Index - RAG Special ๐ŸŒŸ\n\n

\n \n

\n\n

\n https://galileo.ai/hallucination-index\n

\n\n# About the Index\n\n

\n \n

\n\n# Attributes Tested\n\nThere were two key LLM attributes we wanted to test as part of this Index - context length and open vs. closed-source.\n\n## Context Length\n\nWith the rising popularity of RAG, we wanted to see how context length affects model performance. Providing an LLM with context data is akin to giving a student a cheat sheet for an open-book exam. We tested three scenarios:\n\n| Context Length | Task Description |\n| -------------- | ----------- |\n| Short Context | Provide the LLM with < 5k tokens of context data, equivalent to a few pages of information. |\n| Medium Context | Provide the LLM with 5k - 25k tokens of context data, equivalent to a book chapter. |\n| Long Context | Provide the LLM with 40k - 100k tokens of context data, equivalent to an entire book. |\n\n## Open vs. Closed Source\n\nThe open-source vs. closed-source software debate has waged on since the Free Software Movement (FSM) in the late 1980s. This debate has reached a fever pitch during the LLM Arms Race. The assumption is closed-source LLMs, with their access to proprietary training data, will perform better, but we wanted to put this assumption to the test.\n\n## Prompting Techniques\n\nWe experimented with a prompting technique known as Chain-of-Note, which has shown promise for enhancing performance in short-context scenarios, to see if it similarly benefits medium and long contexts.\n\n# Models Evaluated\n\nWe tested 22 models, 10 closed-source models and 12 open-source models, from leading foundation model brands like OpenAI, Anthropic, Meta, Google, Mistral, and more.\n\n

\n \n

\n\n# Major Trends\n\n

\n \n

\n\n# Overall Winners\n\n

\n \n

\n\n# Short Context RAG Insights\n\n

\n \n

\n\n# Medium Context RAG Insights\n\n

\n \n

\n\n# Long Context RAG Insights\n\n

\n \n

\n\n# Methodology\n\n## Short Context RAG (SCR)\n\nWe evaluated SCR using a rigorous set of datasets to test the model's robustness in handling short contexts. One of our key methodologies was Chainpoll with GPT-4o. This involves polling the model multiple times using a chain of thought technique, allowing us to:\n\n1. Quantify potential hallucinations.\n2. Offer context-based explanations, a crucial feature for RAG systems.\n\n## Medium and Long Context RAG (MCR & LCR)\n\nOur focus here was on assessing modelsโ€™ ability to comprehensively understand extensive texts in medium and long contexts. The procedure involved:\n\n- Extracting text from 10,000 recent documents of a company.\n- Dividing the text into chunks and designating one as the \"needle chunk.\"\n- Constructing retrieval questions answerable using the needle chunk embedded in the context.\n\n### Context Lengths Evaluated\n\n- **Medium**: 5k, 10k, 15k, 20k, 25k tokens\n- **Long**: 40k, 60k, 80k, 100k tokens\n\n### Task Design Considerations\n\n1. All text in context must be from a single domain.\n2. Responses should be correct even with short context, confirming the influence of longer contexts.\n3. Questions should not be answerable from pre-training memory or general knowledge.\n4. Measure the influence of information position by keeping everything constant except the location of the needle.\n5. Avoid standard datasets to prevent test leakage.\n\n### Effect of Prompting Technique on Performance\n\nWe experimented with a prompting technique known as Chain-of-Note, which has shown promise for enhancing performance in short-context scenarios, to see if it similarly benefits medium and long contexts.\n\n### Evaluat","github_created_at":"2023-11-15T15:34:16+00:00","created_at":"2026-07-11T12:02:47.845529+00:00","updated_at":"2026-07-29T00:00:34.962834+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"hallucinations","name":"hallucinations"},{"slug":"large-language-models","name":"large language models"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"openai","name":"openai"},{"slug":"rag","name":"rag"},{"slug":"retrieval-augmented-generation","name":"retrieval-augmented-generation"}],"trust":{"provenance":{"is_fork":false,"github_id":719164408,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-29T00:00:34.222Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":365,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:02:49.645Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-29T00:00:34.694Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use when you need to ensure accuracy in short-context tasks, as it tests models like Chain-of-Note prompting techniques specifically for such scenarios.","Choose this tool if you wish to compare both open-source and closed-source LLMs; its methodology assesses the performance gap between different source types under similar test parameters."],"when_not_to_use":["Avoid using Hallucination-Index when your application requires real-time evaluation of hallucinations, as it focuses on predefined tests rather than live model performance.","Do not rely solely on this index if your primary concern is the latest updates to LLM models; its data might not reflect recent improvements in models or the introduction of new ones."],"source":"enrich:decision_facts","observed_at":"2026-07-17T00:15:27.516Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Hallucination-Index helps users identify LLMs with the lowest propensity for factual errors across varying context lengths and source types."}]}}