{"data":{"slug":"akariasai-evidentiality-qa","name":"evidentiality_qa","tagline":"Evidentiality-guided Generator for Knowledge-Intensive NLP Tasks","github_url":"https://github.com/AkariAsai/evidentiality_qa","owner":"AkariAsai","repo":"evidentiality_qa","owner_avatar_url":"https://avatars.githubusercontent.com/u/16631193?v=4","primary_language":"Python","stars":44,"forks":0,"topics":[],"archived":false,"github_pushed_at":"2022-12-25T21:36:20+00:00","maintenance_label":"Dormant","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/akariasai-evidentiality-qa","markdown_url":"https://www.graphcanon.com/tools/akariasai-evidentiality-qa.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/akariasai-evidentiality-qa","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=akariasai-evidentiality-qa","description":"The official implemetation of \"Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks\" (NAACL 2022).","homepage_url":null,"license":"MIT","open_issues":2,"watchers":2,"ai_summary":"Implementation of a model that integrates evidentiality into training for knowledge-intensive NLP tasks, including open question answering, fact verification, and knowledge-enhanced dialogue.","readme_excerpt":"# Evidentiality-guided Generator\nThis is the official implementation of the following paper: Akari Asai, Matt Gardner and Hannaneh Hajishirzi. [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://aclanthology.org/2022.naacl-main.162/). In Proc. NAACL. 2021. \n\nIn this paper, we introduce **Evidentiality-guided Generator**, which incorporates evidentiality of passages---whether a passage contains correct evidence to support the output---into training the generator via multi-task learning of answer generation and evidentiality prediction for retrieval-augmented generation. Experimental results show large improvements across three knowledge intensive tasks: open question answering, fact verification and knowledge-enhanced dialogue. \n\n\n## Directories\n- [`evi_gen`](evi_gen): codes for our evidentiality-guided generator model. The implementation is built upon [Fusion-in-Decoder (Izacard and Grave, 2020)](https://github.com/facebookresearch/FiD).\n\n- [`mining`](mining): code for our evidentiality labeling model used to obtain silver evidentiality data.\n\nPlease see the training and evaluation details in each directories. \n\n## Data and Models\n### Retrieved data (train / dev / test )\nWe release the DPR retrieved results and the results with our silver evidentiality labels. All of the data can be downloaded from [here](https://drive.google.com/drive/folders/1PA4NEJr3W1JXNvofJYBlTo5nyyGkMqRL?usp=sharing).      \n- [`evidentiality_dpr.zip`](https://drive.google.com/file/d/1BnWMB9XS63HPRVq7eWYJ3h4JvVsfr6-5/view?usp=sharing) includes the retrieval results with our newly mined silver evidentiality labels for train sets for each target dataset. For each query, we include top 20 passages. \n- [`eval_dpr.zip`](https://drive.google.com/file/d/1hXr04jaGuapqpcaROsM3xsSyQ100F6zd/view?usp=sharing) includes the retrieval results for dev / test sets for each target dataset. \n\n### Fine-tuned models\nYou can download the fine-tuned models from the google drive repositories. \n\n- [NQ Open](https://drive.google.com/file/d/16bio8wIvbIj7OmWqaiBzGFEHBFB6AlY6/view?usp=sharing)\n- [TriviaQA unfiltered](https://drive.google.com/file/d/1YskriNt9LMUUZnGqSbhlE4Cyiz8CSaPG/view?usp=sharing)\n- [FaVIQ unfiltered](https://drive.google.com/file/d/1OnFxXzJTWbu_rWylDmS0P8gXHMZJPGbW/view?usp=sharing)\n- [FEVER unfiltered](https://drive.google.com/file/d/19qa1vr4ng_GAqGcygr5ErCuSh5fnqhpr/view?usp=sharing)\n- [WoW](https://drive.google.com/file/d/10RsFfGgzsSC9MOb3Csoc1ztUp0NJSoGy/view?usp=sharing)\n\n## Evaluations\nTo reproduce the original results, you can go to `evi_gen` directory and then run the command below:\n\n```\nCUDA_VISIBLE_DEVICES=0 python test_reader.py \\\n    --model_path model/nq_ours \\\n    --eval_data data/nq_test.json \\\n    --per_gpu_batch_size 48 \\\n    --n_context 20 \\\n    --name sanity_nq_test \\\n    --checkpoint_dir checkpoint \\\n    --n_gpus 1 \\\n    --write_results\n```\n\nFor WoW, please set the `--metric f1`\n\n\n\n## Training\n### Overview of Training\nOur evidentiality-guided generator will conduct a multi-task learning of evidentiality prediction and generation.      \nTo supervised this learning, we need to obtain *silver* evidentiality data.   \n\nOur training procedures are as follows:\n1. Training a base Fusion-in-Decoder model (*base generator*)\n2. Run leave-one-out generation approach to collect training data for evidentiality labeling model (M) using the base generator.\n3. Train M using data from step 2.\n4. Run M on all of the passages included in training data for a evidentiality-guided generator to obtain silver evidentiality labels. \n5. Train the evidentiality-guided generator with the multi-task loss.\n\nSee more detailed instructions in the `evi_gen` and `mining` directories.  \n\n### Training our evidentiality generator\nIf you want to quickly start training our evidentiality-guided generator, we provide the resulting training data [here](https://drive.google.com/file/d/1BnWMB9XS63HPRVq7eWYJ3h4JvVsfr6-5/view?usp=sharing).         \nTo train our evide","github_created_at":"2021-12-16T09:12:05+00:00","created_at":"2026-07-11T23:07:45.319039+00:00","updated_at":"2026-09-20T04:29:11.109283+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"evidentiality","name":"evidentiality"},{"slug":"fine-tuned-models","name":"fine-tuned models"},{"slug":"knowledge-intensive-tasks","name":"knowledge-intensive tasks"},{"slug":"multi-task-learning","name":"multi-task learning"},{"slug":"nlp","name":"nlp"},{"slug":"open-question-answering","name":"open question answering"},{"slug":"retrieval-augmented-generation","name":"retrieval-augmented generation"},{"slug":"silver-evidentiality-data","name":"silver evidentiality data"}],"trust":{"provenance":{"is_fork":false,"github_id":438927243,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:13:34.000Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1362,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:13:36.380Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:13:32.872Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-18T08:13:32.872Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-18T08:13:32.872Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["The model requires access to silver evidentiality data for training.","Training involves a multi-step process including training a base Fusion-in-Decoder model and a separate evidentiality labeling model."],"min_ram_gb":8,"requires_docker":false},"constraints":{"min_ram_gb":8,"requires_docker":false},"when_to_use":["Use evidentiality_qa when you need to improve performance on knowledge-intensive NLP tasks that require the model to generate answers based on evidence from retrieved passages.","Consider evidentiality_qa for tasks where the quality of evidence supporting the generated output is critical, such as in fact verification or knowledge-enhanced dialogue systems."],"when_not_to_use":["Avoid using evidentiality_qa if your task does not involve knowledge-intensive NLP tasks that require evidence-based reasoning.","Do not use evidentiality_qa if you lack the necessary silver evidentiality data for training, as the model relies on this data to predict evidentiality and generate answers."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:30:12.480Z"},"constraint_facets":{"min_ram_gb":8,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Min 8 GB RAM; The model requires access to silver evidentiality data for training.; Training involves a multi-step process including training a base Fusion-in-Decoder model and a separate evidentiality labeling model."},{"label":"Adopt for","value":"Evidentiality_qa is a Python-based model that integrates evidentiality into training for knowledge-intensive NLP tasks, such as open question answering, fact verification, and knowledge-enhanced dialogue, using multi-tas"},{"label":"License detail","value":"MIT License"}]}}