{"data":{"slug":"akariasai-evidentiality-qa","name":"evidentiality_qa","tagline":"The official implemetation of \"Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks\" (NAACL 2022).","github_url":"https://github.com/AkariAsai/evidentiality_qa","owner":"AkariAsai","repo":"evidentiality_qa","owner_avatar_url":"https://avatars.githubusercontent.com/u/16631193?v=4","primary_language":"Python","stars":44,"forks":0,"topics":[],"archived":false,"github_pushed_at":"2022-12-25T21:36:20+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/akariasai-evidentiality-qa","markdown_url":"https://www.graphcanon.com/tools/akariasai-evidentiality-qa.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/akariasai-evidentiality-qa","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=akariasai-evidentiality-qa","description":"The official implemetation of \"Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks\" (NAACL 2022).","homepage_url":null,"license":"MIT","open_issues":2,"watchers":2,"ai_summary":null,"readme_excerpt":"# Evidentiality-guided Generator\nThis is the official implementation of the following paper: Akari Asai, Matt Gardner and Hannaneh Hajishirzi. [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://aclanthology.org/2022.naacl-main.162/). In Proc. NAACL. 2021. \n\nIn this paper, we introduce **Evidentiality-guided Generator**, which incorporates evidentiality of passages---whether a passage contains correct evidence to support the output---into training the generator via multi-task learning of answer generation and evidentiality prediction for retrieval-augmented generation. Experimental results show large improvements across three knowledge intensive tasks: open question answering, fact verification and knowledge-enhanced dialogue. \n\n\n## Directories\n- [`evi_gen`](evi_gen): codes for our evidentiality-guided generator model. The implementation is built upon [Fusion-in-Decoder (Izacard and Grave, 2020)](https://github.com/facebookresearch/FiD).\n\n- [`mining`](mining): code for our evidentiality labeling model used to obtain silver evidentiality data.\n\nPlease see the training and evaluation details in each directories. \n\n## Data and Models\n### Retrieved data (train / dev / test )\nWe release the DPR retrieved results and the results with our silver evidentiality labels. All of the data can be downloaded from [here](https://drive.google.com/drive/folders/1PA4NEJr3W1JXNvofJYBlTo5nyyGkMqRL?usp=sharing).      \n- [`evidentiality_dpr.zip`](https://drive.google.com/file/d/1BnWMB9XS63HPRVq7eWYJ3h4JvVsfr6-5/view?usp=sharing) includes the retrieval results with our newly mined silver evidentiality labels for train sets for each target dataset. For each query, we include top 20 passages. \n- [`eval_dpr.zip`](https://drive.google.com/file/d/1hXr04jaGuapqpcaROsM3xsSyQ100F6zd/view?usp=sharing) includes the retrieval results for dev / test sets for each target dataset. \n\n### Fine-tuned models\nYou can download the fine-tuned models from the google drive repositories. \n\n- [NQ Open](https://drive.google.com/file/d/16bio8wIvbIj7OmWqaiBzGFEHBFB6AlY6/view?usp=sharing)\n- [TriviaQA unfiltered](https://drive.google.com/file/d/1YskriNt9LMUUZnGqSbhlE4Cyiz8CSaPG/view?usp=sharing)\n- [FaVIQ unfiltered](https://drive.google.com/file/d/1OnFxXzJTWbu_rWylDmS0P8gXHMZJPGbW/view?usp=sharing)\n- [FEVER unfiltered](https://drive.google.com/file/d/19qa1vr4ng_GAqGcygr5ErCuSh5fnqhpr/view?usp=sharing)\n- [WoW](https://drive.google.com/file/d/10RsFfGgzsSC9MOb3Csoc1ztUp0NJSoGy/view?usp=sharing)\n\n## Evaluations\nTo reproduce the original results, you can go to `evi_gen` directory and then run the command below:\n\n```\nCUDA_VISIBLE_DEVICES=0 python test_reader.py \\\n    --model_path model/nq_ours \\\n    --eval_data data/nq_test.json \\\n    --per_gpu_batch_size 48 \\\n    --n_context 20 \\\n    --name sanity_nq_test \\\n    --checkpoint_dir checkpoint \\\n    --n_gpus 1 \\\n    --write_results\n```\n\nFor WoW, please set the `--metric f1`\n\n\n\n## Training\n### Overview of Training\nOur evidentiality-guided generator will conduct a multi-task learning of evidentiality prediction and generation.      \nTo supervised this learning, we need to obtain *silver* evidentiality data.   \n\nOur training procedures are as follows:\n1. Training a base Fusion-in-Decoder model (*base generator*)\n2. Run leave-one-out generation approach to collect training data for evidentiality labeling model (M) using the base generator.\n3. Train M using data from step 2.\n4. Run M on all of the passages included in training data for a evidentiality-guided generator to obtain silver evidentiality labels. \n5. Train the evidentiality-guided generator with the multi-task loss.\n\nSee more detailed instructions in the `evi_gen` and `mining` directories.  \n\n### Training our evidentiality generator\nIf you want to quickly start training our evidentiality-guided generator, we provide the resulting training data [here](https://drive.google.com/file/d/1BnWMB9XS63HPRVq7eWYJ3h4JvVsfr6-5/view?usp=sharing).         \nTo train our evide","github_created_at":"2021-12-16T09:12:05+00:00","created_at":"2026-07-11T23:07:45.319039+00:00","updated_at":"2026-07-11T23:07:51.432864+00:00","categories":[{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":438927243,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:07:48.644Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1294,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:07:49.125Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:07:48.405Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:07:48.405Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T23:07:48.405Z"}}}}