--- title: "LLM-Knowledge-Conflict" type: "tool" slug: "osu-nlp-group-llm-knowledge-conflict" canonical_url: "https://www.graphcanon.com/tools/osu-nlp-group-llm-knowledge-conflict" github_url: "https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict" homepage_url: null stars: 84 forks: 4 primary_language: "Python" license: "Apache-2.0" archived: false categories: ["llm-frameworks", "evaluation-observability"] tags: ["conflicting-evidence-handling", "language-model-behavior-analysis", "knowledge-conflicts", "parametric-memory", "conflict-resolution", "datasets-for-evaluation", "data-retrieval"] updated_at: "2026-07-12T06:44:40.200769+00:00" --- # LLM-Knowledge-Conflict > [ICLR'24 Spotlight] Revealing the Behavior of Large Language Models in Knowledge Conflicts This project provides datasets and tools aimed at understanding how large language models handle knowledge conflict scenarios through a suite of parametric memory techniques. ## Facts - Repository: https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict - Stars: 84 · Forks: 4 · Open issues: 1 · Watchers: 3 - Primary language: Python - License: Apache-2.0 - Last pushed: 2024-04-12T06:38:36+00:00 ## Trust & health _Signals computed from public GitHub metadata. Not a security guarantee._ - Maintenance: Dormant (computed 2026-07-11T23:06:04.189Z) - Security scan: No lockfile (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-11T23:06:04.690Z - Full report: [trust report](/tools/osu-nlp-group-llm-knowledge-conflict/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/osu-nlp-group-llm-knowledge-conflict/trust) ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) - [Evaluation & Observability](/categories/evaluation-observability.md) ## Tags conflicting evidence handling, language model behavior analysis, knowledge conflicts, parametric memory, conflict resolution, datasets for evaluation, data retrieval ## Category neighbours (exploratory) _Same-category tools for discovery only - not curated alternatives. Cap shown at six._ - [RAG_Techniques](/tools/nirdiamant-rag-techniques.md) - Showcases advanced techniques for Retrieval-Augmented Generation (RAG) systems with detailed notebook tutorials. (★ 28,465) [Active] - [WeKnora](/tools/tencent-weknora.md) - Open-source LLM knowledge platform for creating a queryable RAG, autonomous reasoning agent, and self-maintaining Wiki. (★ 18,122) [Very active] - [open-llms](/tools/eugeneyan-open-llms.md) - A list of open LLMs available for commercial use. (★ 12,825) [Dormant] - [LLMSurvey](/tools/rucaibox-llmsurvey.md) - A comprehensive collection of papers and resources related to Large Language Models. (★ 12,187) [Dormant] - [LLMsPracticalGuide](/tools/mooler0410-llmspracticalguide.md) - A curated list of practical guide resources of LLMs (★ 10,200) [Slowing] - [awesome-LLM-resources](/tools/wangrongsheng-awesome-llm-resources.md) - 🧑🚀 全世界最好的LLM资料总结（多模态生成、Agent、辅助编程、AI审稿、数据处理、模型训练、模型推理、o1 模型、MCP、小语言模型、视觉语言模型） | Summary of the world's best LLM resources. (★ 8,668) [Very active] _+ 2 more not listed._ ## Adoption goal LLM-Knowledge-Conflict provides specific datasets and tools to understand how large language models handle knowledge conflicts by using parametric memory techniques. ## README (excerpt) _Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._ ````text # [ICLR'24 Spotlight] LLM-knowledge-conflict ### ConflictQA We provide the conflictQA based on different large language models, which utilizes large language models guided parametric memory. The data is available at [conflictQA](conflictQA) foloder. This folder contains the data for both POPQA and STRATEGYQA ```json {"question": "What is George Rankin's occupation?", "popularity": 142, "ground_truth": ["politician", "political leader", "political figure", "polit.", "pol"], "memory_answer": "George Rankin's occupation is a professional photographer.", "parametric_memory": "As a professional photographer, George Rankin...", "counter_answer": "George Rankin's occupation is political figure.", "counter_memory": "George Rankin has been actively involved in politics for over a decade...", "parametric_memory_aligned_evidence": "George Rankin has a website showcasing his photography portfolio...", "counter_memory_aligned_evidence": "George Rankin Major General George James Rankin..."} ``` - "question": The question in natural language - "popularity": The monthly page views on Wikipedia for the given question - "ground_truth": The factual answer to the question, which may include multiple possible answers - "memory_answer": The answer provided by the LLM to the question - "parametric_memory": The supportive evidence from LLM's parametric memory for the answer - "counter_answer": The answer contradicting the "memory_answer" - "counter_memory": The generation-based evidence supporting the counter_answer - "parametric_memory_aligned_evidence": Additional evidence supporting the "memory_answer", which could be generated or derived from Wikipedia/human annotation - "counter_memory_aligned_evidence": Additional evidence supporting the "counter_answer", either generated or sourced from Wikipedia/human annotation We also release our dataset at: Huggingface datasets: https://huggingface.co/datasets/osunlp/ConflictQA (more details can be found on the dataset page) ```python #loading dataset from datasets import load_dataset # you can choose dataset "ConflictQA-popQA-[PLACEHOLDER]", and the [PLACEHOLDER] is in ["chatgpt","gpt4","palm2","llama2-7b","llama2-70b","qwen7b","vicuna7b","vicuna33b"]. dataset = load_dataset("osunlp/ConflictQA",'ConflictQA-popQA-chatgpt') ``` Code is available in [code](code) foloder. ### Citation If our paper or related resources prove valuable to your research, we kindly ask for citation. Please feel free to contact us with any inquiries. ```bib @inproceedings{ xie2024knowledgeconflict, title={Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts}, author={Jian Xie and Kai Zhang and Jiangjie Chen and Renze Lou and Yu Su}, booktitle={The Twelfth International Conference on Learning Representations}, year={2024}, url={https://openreview.net/forum?id=auKAUJZMO6} } ``` ```` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/osu-nlp-group-llm-knowledge-conflict`](/api/graphcanon/tools/osu-nlp-group-llm-knowledge-conflict) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_