hallucination-index
Initiative to evaluate and rank popular LLMs based on hallucination propensity
GraphCanon updated 3w · GitHub synced 3w
Decision brief
Hallucination-Index helps users identify LLMs with the lowest propensity for factual errors across varying context lengths and source types.
Good fit when
- Use when you need to ensure accuracy in short-context tasks, as it tests models like Chain-of-Note prompting techniques specifically for such scenarios.
- Choose this tool if you wish to compare both open-source and closed-source LLMs; its methodology assesses the performance gap between different source types under similar test parameters.
Avoid when
- Avoid using Hallucination-Index when your application requires real-time evaluation of hallucinations, as it focuses on predefined tests rather than live model performance.
- Do not rely solely on this index if your primary concern is the latest updates to LLM models; its data might not reflect recent improvements in models or the introduction of new ones.
Observed Jul 17, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (365d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/rungalileo/hallucination-indexSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
The repository evaluates the tendency of large language models (LLMs) to generate factual errors, known as 'hallucinations,' by testing model attributes such as context length and source openness. It includes a comprehensive assessment of 22 open-source and closed-source models using various prompting techniques across short, medium, and long contexts.
Capability facts
No sourced capability facts yet. Facts appear after ingest scans repo manifests (Dockerfile, package.json, MCP configs).
Categories
Tags
README
🌟 LLM Hallucination Index - RAG Special 🌟
https://galileo.ai/hallucination-index
About the Index
Attributes Tested
There were two key LLM attributes we wanted to test as part of this Index - context length and open vs. closed-source.
Context Length
With the rising popularity of RAG, we wanted to see how context length affects model performance. Providing an LLM with context data is akin to giving a student a cheat sheet for an open-book exam. We tested three scenarios:
| Context Length | Task Description |
|---|---|
| Short Context | Provide the LLM with < 5k tokens of context data, equivalent to a few pages of information. |
| Medium Context | Provide the LLM with 5k - 25k tokens of context data, equivalent to a book chapter. |
| Long Context | Provide the LLM with 40k - 100k tokens of context data, equivalent to an entire book. |
Open vs. Closed Source
The open-source vs. closed-source software debate has waged on since the Free Software Movement (FSM) in the late 1980s. This debate has reached a fever pitch during the LLM Arms Race. The assumption is closed-source LLMs, with their access to proprietary training data, will perform better, but we wanted to put this assumption to the test.
Prompting Techniques
We experimented with a prompting technique known as Chain-of-Note, which has shown promise for enhancing performance in short-context scenarios, to see if it similarly benefits medium and long contexts.
Models Evaluated
We tested 22 models, 10 closed-source models and 12 open-source models, from leading foundation model brands like OpenAI, Anthropic, Meta, Google, Mistral, and more.
Major Trends
Overall Winners
Short Context RAG Insights
Medium Context RAG Insights
Long Context RAG Insights
Methodology
Short Context RAG (SCR)
We evaluated SCR using a rigorous set of datasets to test the model's robustness in handling short contexts. One of our key methodologies was Chainpoll with GPT-4o. This involves polling the model multiple times using a chain of thought technique, allowing us to:
- Quantify potential hallucinations.
- Offer context-based explanations, a crucial feature for RAG systems.
Medium and Long Context RAG (MCR & LCR)
Our focus here was on assessing models’ ability to comprehensively understand extensive texts in medium and long contexts. The procedure involved:
- Extracting text from 10,000 recent documents of a company.
- Dividing the text into chunks and designating one as the "needle chunk."
- Constructing retrieval questions answerable using the needle chunk embedded in the context.
Context Lengths Evaluated
- Medium: 5k, 10k, 15k, 20k, 25k tokens
- Long: 40k, 60k, 80k, 100k tokens
Task Design Considerations
- All text in context must be from a single domain.
- Responses should be correct even with short context, confirming the influence of longer contexts.
- Questions should not be answerable from pre-training memory or general knowledge.
- Measure the influence of information position by keeping everything constant except the location of the needle.
- Avoid standard datasets to prevent test leakage.
Effect of Prompting Technique on Performance
We experimented with a prompting technique known as Chain-of-Note, which has shown promise for enhancing performance in short-context scenarios, to see if it similarly benefits medium and long contexts.
Evaluat
For agents
This page has a .md twin and JSON over the API.