{"data":{"slug":"osu-nlp-group-llm-knowledge-conflict","name":"LLM-Knowledge-Conflict","tagline":"[ICLR'24 Spotlight] Revealing the Behavior of Large Language Models in Knowledge Conflicts","github_url":"https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict","owner":"OSU-NLP-Group","repo":"LLM-Knowledge-Conflict","owner_avatar_url":"https://avatars.githubusercontent.com/u/92067480?v=4","primary_language":"Python","stars":84,"forks":4,"topics":[],"archived":false,"github_pushed_at":"2024-04-12T06:38:36+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/osu-nlp-group-llm-knowledge-conflict","markdown_url":"https://www.graphcanon.com/tools/osu-nlp-group-llm-knowledge-conflict.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/osu-nlp-group-llm-knowledge-conflict","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=osu-nlp-group-llm-knowledge-conflict","description":"[ICLR'24 Spotlight] \"Adaptive Chameleon  or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts\"","homepage_url":null,"license":"Apache-2.0","open_issues":1,"watchers":3,"ai_summary":"This project provides datasets and tools aimed at understanding how large language models handle knowledge conflict scenarios through a suite of parametric memory techniques.","readme_excerpt":"# [ICLR'24 Spotlight] LLM-knowledge-conflict\n### ConflictQA\n\nWe provide the conflictQA  based on different large language models, which utilizes large language models guided parametric memory.\n\nThe data is available at [conflictQA](conflictQA) foloder. This folder contains the data for both POPQA and STRATEGYQA\n\n```json\n{\"question\": \"What is George Rankin's occupation?\", \"popularity\": 142, \"ground_truth\": [\"politician\", \"political leader\", \"political figure\", \"polit.\", \"pol\"], \"memory_answer\": \"George Rankin's occupation is a professional photographer.\", \"parametric_memory\": \"As a professional photographer, George Rankin...\", \"counter_answer\": \"George Rankin's occupation is political figure.\", \"counter_memory\": \"George Rankin has been actively involved in politics for over a decade...\", \"parametric_memory_aligned_evidence\": \"George Rankin has a website showcasing his photography portfolio...\", \"counter_memory_aligned_evidence\": \"George Rankin Major General George James Rankin...\"}\n```\n\n- \"question\": The question in natural language\n- \"popularity\": The monthly page views on Wikipedia for the given question\n- \"ground_truth\": The factual answer to the question, which may include multiple possible answers\n- \"memory_answer\": The answer provided by the LLM to the question\n- \"parametric_memory\": The supportive evidence from LLM's parametric memory for the answer\n- \"counter_answer\": The answer contradicting the \"memory_answer\"\n- \"counter_memory\": The generation-based evidence supporting the counter_answer\n- \"parametric_memory_aligned_evidence\": Additional evidence supporting the \"memory_answer\", which could be generated or derived from Wikipedia/human annotation\n- \"counter_memory_aligned_evidence\": Additional evidence supporting the \"counter_answer\", either generated or sourced from Wikipedia/human annotation\n\n\n\nWe also release our dataset at: Huggingface datasets: https://huggingface.co/datasets/osunlp/ConflictQA (more details can be found on the dataset page)\n\n```python\n#loading dataset\nfrom datasets import load_dataset\n# you can choose dataset \"ConflictQA-popQA-[PLACEHOLDER]\", and the [PLACEHOLDER] is in [\"chatgpt\",\"gpt4\",\"palm2\",\"llama2-7b\",\"llama2-70b\",\"qwen7b\",\"vicuna7b\",\"vicuna33b\"].\ndataset = load_dataset(\"osunlp/ConflictQA\",'ConflictQA-popQA-chatgpt')\n```\n\nCode is available in [code](code) foloder.\n\n### Citation\n\nIf our paper or related resources prove valuable to your research, we kindly ask for citation. Please feel free to contact us with any inquiries.\n\n```bib\n@inproceedings{\nxie2024knowledgeconflict,\ntitle={Adaptive Chameleon  or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts},\nauthor={Jian Xie and Kai Zhang and Jiangjie Chen and Renze Lou and Yu Su},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=auKAUJZMO6}\n}\n```","github_created_at":"2023-05-22T02:18:49+00:00","created_at":"2026-07-11T23:06:01.616306+00:00","updated_at":"2026-07-12T06:44:40.200769+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"conflicting-evidence-handling","name":"conflicting evidence handling"},{"slug":"language-model-behavior-analysis","name":"language model behavior analysis"},{"slug":"knowledge-conflicts","name":"knowledge conflicts"},{"slug":"parametric-memory","name":"parametric memory"},{"slug":"conflict-resolution","name":"conflict resolution"},{"slug":"datasets-for-evaluation","name":"datasets for evaluation"},{"slug":"data-retrieval","name":"data retrieval"}],"trust":{"provenance":{"is_fork":false,"github_id":643716093,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:06:04.189Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":820,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:06:04.690Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T06:44:18.144Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-12T06:44:18.144Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-12T06:44:18.144Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you want to evaluate the robustness of a large language model's responses in scenarios where conflicting information is available.","To analyze and improve the consistency and accuracy of responses from multiple pre-trained LLMs like ChatGPT, GPT-4, etc., by using parametric memory aligned evidence."],"when_not_to_use":["If your objective is to train new large language models rather than evaluate existing ones under specific scenarios.","When you require a general-purpose natural language processing toolkit that includes tasks beyond the scope of knowledge conflict evaluation."],"source":"enrich:decision_facts","observed_at":"2026-07-12T06:44:39.759Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"LLM-Knowledge-Conflict provides specific datasets and tools to understand how large language models handle knowledge conflicts by using parametric memory techniques."}]}}