{"data":{"slug":"osu-nlp-group-llm-knowledge-conflict","name":"LLM-Knowledge-Conflict","tagline":"Examines behavior of large language models in knowledge conflicts using ConflictQA dataset","github_url":"https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict","owner":"OSU-NLP-Group","repo":"LLM-Knowledge-Conflict","owner_avatar_url":"https://avatars.githubusercontent.com/u/92067480?v=4","primary_language":"Python","stars":87,"forks":4,"topics":[],"archived":false,"github_pushed_at":"2024-04-12T06:38:36+00:00","maintenance_label":"Dormant","stars_delta_30d":3,"url":"https://www.graphcanon.com/tools/osu-nlp-group-llm-knowledge-conflict","markdown_url":"https://www.graphcanon.com/tools/osu-nlp-group-llm-knowledge-conflict.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/osu-nlp-group-llm-knowledge-conflict","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=osu-nlp-group-llm-knowledge-conflict","description":"[ICLR'24 Spotlight] \"Adaptive Chameleon  or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts\"","homepage_url":null,"license":"Apache-2.0","open_issues":1,"watchers":3,"ai_summary":"This repository contains ConflictQA, a dataset and evaluation framework for assessing how large language models handle knowledge conflicts. It includes data and code for analyzing model responses and their alignment with ground truth and conflicting evidence.","readme_excerpt":"# [ICLR'24 Spotlight] LLM-knowledge-conflict\n### ConflictQA\n\nWe provide the conflictQA  based on different large language models, which utilizes large language models guided parametric memory.\n\nThe data is available at [conflictQA](conflictQA) foloder. This folder contains the data for both POPQA and STRATEGYQA\n\n```json\n{\"question\": \"What is George Rankin's occupation?\", \"popularity\": 142, \"ground_truth\": [\"politician\", \"political leader\", \"political figure\", \"polit.\", \"pol\"], \"memory_answer\": \"George Rankin's occupation is a professional photographer.\", \"parametric_memory\": \"As a professional photographer, George Rankin...\", \"counter_answer\": \"George Rankin's occupation is political figure.\", \"counter_memory\": \"George Rankin has been actively involved in politics for over a decade...\", \"parametric_memory_aligned_evidence\": \"George Rankin has a website showcasing his photography portfolio...\", \"counter_memory_aligned_evidence\": \"George Rankin Major General George James Rankin...\"}\n```\n\n- \"question\": The question in natural language\n- \"popularity\": The monthly page views on Wikipedia for the given question\n- \"ground_truth\": The factual answer to the question, which may include multiple possible answers\n- \"memory_answer\": The answer provided by the LLM to the question\n- \"parametric_memory\": The supportive evidence from LLM's parametric memory for the answer\n- \"counter_answer\": The answer contradicting the \"memory_answer\"\n- \"counter_memory\": The generation-based evidence supporting the counter_answer\n- \"parametric_memory_aligned_evidence\": Additional evidence supporting the \"memory_answer\", which could be generated or derived from Wikipedia/human annotation\n- \"counter_memory_aligned_evidence\": Additional evidence supporting the \"counter_answer\", either generated or sourced from Wikipedia/human annotation\n\n\n\nWe also release our dataset at: Huggingface datasets: https://huggingface.co/datasets/osunlp/ConflictQA (more details can be found on the dataset page)\n\n```python\n#loading dataset\nfrom datasets import load_dataset\n# you can choose dataset \"ConflictQA-popQA-[PLACEHOLDER]\", and the [PLACEHOLDER] is in [\"chatgpt\",\"gpt4\",\"palm2\",\"llama2-7b\",\"llama2-70b\",\"qwen7b\",\"vicuna7b\",\"vicuna33b\"].\ndataset = load_dataset(\"osunlp/ConflictQA\",'ConflictQA-popQA-chatgpt')\n```\n\nCode is available in [code](code) foloder.\n\n### Citation\n\nIf our paper or related resources prove valuable to your research, we kindly ask for citation. Please feel free to contact us with any inquiries.\n\n```bib\n@inproceedings{\nxie2024knowledgeconflict,\ntitle={Adaptive Chameleon  or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts},\nauthor={Jian Xie and Kai Zhang and Jiangjie Chen and Renze Lou and Yu Su},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=auKAUJZMO6}\n}\n```","github_created_at":"2023-05-22T02:18:49+00:00","created_at":"2026-07-11T23:06:01.616306+00:00","updated_at":"2026-09-20T04:28:56.85808+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"conflictqa","name":"conflictqa"},{"slug":"dataset","name":"dataset"},{"slug":"iclr-24","name":"iclr'24"},{"slug":"knowledge-conflict","name":"knowledge conflict"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"parametric-memory","name":"parametric memory"}],"trust":{"provenance":{"is_fork":false,"github_id":643716093,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:11:32.312Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":889,"last_release_at":null,"stars_delta_30d":3,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:11:34.611Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:11:31.120Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-18T08:11:31.120Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:11:31.120Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"The tool is free to use under the Apache-2.0 license, but users may need to consider costs associated with running large language models."},"requirements":{"notes":["Python environment is required.","The dataset can be loaded using Huggingface datasets."],"min_ram_gb":8,"requires_docker":false},"constraints":{"min_ram_gb":8,"pricing_model":"freemium","requires_docker":false},"when_to_use":["When you need to assess how large language models handle conflicting information and their alignment with ground truth.","If you are researching or developing large language models and want to understand their behavior in scenarios where conflicting evidence is presented.","For evaluating the robustness and reliability of large language models in situations where they might encounter conflicting information."],"when_not_to_use":["If you are looking for a tool that focuses on general performance metrics of large language models without considering knowledge conflicts.","When your primary interest is in the speed or efficiency of large language models rather than their handling of conflicting information.","If your project does not require an in-depth analysis of how models respond to conflicting evidence and align with ground truth."],"source":"enrich:decision_facts","observed_at":"2026-09-18T11:53:39.738Z"},"constraint_facets":{"min_ram_gb":8,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium - The tool is free to use under the Apache-2.0 license, but users may need to consider costs associated with running large language models."},{"label":"Requirements","value":"Min 8 GB RAM; Python environment is required.; The dataset can be loaded using Huggingface datasets."},{"label":"Adopt for","value":"LLM-Knowledge-Conflict is a tool for evaluating how large language models handle knowledge conflicts using the ConflictQA dataset. It provides a framework for analyzing model responses and their alignment with ground-tru"},{"label":"License detail","value":"The tool is licensed under Apache-2.0, allowing for free use, modification, and distribution."}]}}