{"data":{"slug":"rungalileo-hallucination-index","name":"hallucination-index","tagline":"Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate.","github_url":"https://github.com/rungalileo/hallucination-index","owner":"rungalileo","repo":"hallucination-index","owner_avatar_url":"https://avatars.githubusercontent.com/u/81123343?v=4","primary_language":null,"stars":116,"forks":9,"topics":["hallucinations","large-language-models","llm","llm-evaluation","openai","rag","retrieval-augmented-generation"],"archived":false,"github_pushed_at":"2025-07-28T18:28:09+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/rungalileo-hallucination-index","markdown_url":"https://www.graphcanon.com/tools/rungalileo-hallucination-index.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/rungalileo-hallucination-index","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=rungalileo-hallucination-index","description":"Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate. ","homepage_url":"https://www.rungalileo.io/hallucinationindex","license":null,"open_issues":1,"watchers":5,"ai_summary":null,"readme_excerpt":"# ๐ŸŒŸ LLM Hallucination Index - RAG Special ๐ŸŒŸ\n\n

\n \n

\n\n

\n https://galileo.ai/hallucination-index\n

\n\n# About the Index\n\n

\n \n

\n\n# Attributes Tested\n\nThere were two key LLM attributes we wanted to test as part of this Index - context length and open vs. closed-source.\n\n## Context Length\n\nWith the rising popularity of RAG, we wanted to see how context length affects model performance. Providing an LLM with context data is akin to giving a student a cheat sheet for an open-book exam. We tested three scenarios:\n\n| Context Length | Task Description |\n| -------------- | ----------- |\n| Short Context | Provide the LLM with < 5k tokens of context data, equivalent to a few pages of information. |\n| Medium Context | Provide the LLM with 5k - 25k tokens of context data, equivalent to a book chapter. |\n| Long Context | Provide the LLM with 40k - 100k tokens of context data, equivalent to an entire book. |\n\n## Open vs. Closed Source\n\nThe open-source vs. closed-source software debate has waged on since the Free Software Movement (FSM) in the late 1980s. This debate has reached a fever pitch during the LLM Arms Race. The assumption is closed-source LLMs, with their access to proprietary training data, will perform better, but we wanted to put this assumption to the test.\n\n## Prompting Techniques\n\nWe experimented with a prompting technique known as Chain-of-Note, which has shown promise for enhancing performance in short-context scenarios, to see if it similarly benefits medium and long contexts.\n\n# Models Evaluated\n\nWe tested 22 models, 10 closed-source models and 12 open-source models, from leading foundation model brands like OpenAI, Anthropic, Meta, Google, Mistral, and more.\n\n

\n \n

\n\n# Major Trends\n\n

\n \n

\n\n# Overall Winners\n\n

\n \n

\n\n# Short Context RAG Insights\n\n

\n \n

\n\n# Medium Context RAG Insights\n\n

\n \n

\n\n# Long Context RAG Insights\n\n

\n \n

\n\n# Methodology\n\n## Short Context RAG (SCR)\n\nWe evaluated SCR using a rigorous set of datasets to test the model's robustness in handling short contexts. One of our key methodologies was Chainpoll with GPT-4o. This involves polling the model multiple times using a chain of thought technique, allowing us to:\n\n1. Quantify potential hallucinations.\n2. Offer context-based explanations, a crucial feature for RAG systems.\n\n## Medium and Long Context RAG (MCR & LCR)\n\nOur focus here was on assessing modelsโ€™ ability to comprehensively understand extensive texts in medium and long contexts. The procedure involved:\n\n- Extracting text from 10,000 recent documents of a company.\n- Dividing the text into chunks and designating one as the \"needle chunk.\"\n- Constructing retrieval questions answerable using the needle chunk embedded in the context.\n\n### Context Lengths Evaluated\n\n- **Medium**: 5k, 10k, 15k, 20k, 25k tokens\n- **Long**: 40k, 60k, 80k, 100k tokens\n\n### Task Design Considerations\n\n1. All text in context must be from a single domain.\n2. Responses should be correct even with short context, confirming the influence of longer contexts.\n3. Questions should not be answerable from pre-training memory or general knowledge.\n4. Measure the influence of information position by keeping everything constant except the location of the needle.\n5. Avoid standard datasets to prevent test leakage.\n\n### Effect of Prompting Technique on Performance\n\nWe experimented with a prompting technique known as Chain-of-Note, which has shown promise for enhancing performance in short-context scenarios, to see if it similarly benefits medium and long contexts.\n\n### Evaluat","github_created_at":"2023-11-15T15:34:16+00:00","created_at":"2026-07-11T12:02:47.845529+00:00","updated_at":"2026-07-11T12:02:53.258192+00:00","categories":[{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"hallucinations","name":"hallucinations"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"llm","name":"llm"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"openai","name":"openai"},{"slug":"rag","name":"rag"},{"slug":"retrieval-augmented-generation","name":"retrieval-augmented-generation"}],"trust":{"provenance":{"is_fork":false,"github_id":719164408,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T12:02:48.564Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":347,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:02:49.645Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T12:02:49.408Z"}}}}