{"data":{"slug":"declare-lab-instruct-eval","name":"instruct-eval","tagline":"Code for evaluating instruction-tuned language models like Alpaca and Flan-T5","github_url":"https://github.com/declare-lab/instruct-eval","owner":"declare-lab","repo":"instruct-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/59164695?v=4","primary_language":"Python","stars":552,"forks":45,"topics":["instruct-tuning","llm"],"archived":false,"github_pushed_at":"2024-03-10T05:00:00+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/declare-lab-instruct-eval","markdown_url":"https://www.graphcanon.com/tools/declare-lab-instruct-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/declare-lab-instruct-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=declare-lab-instruct-eval","description":"This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. ","homepage_url":"https://declare-lab.github.io/instruct-eval/","license":"Apache-2.0","open_issues":24,"watchers":9,"ai_summary":"InstructEval offers tools to quantitatively assess the performance of instruction-tuned large language models on unseen tasks, including measures related to safety and problem-solving capabilities.","readme_excerpt":"## :camel: 🍮 📚 InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models\n\n[Paper](https://arxiv.org/abs/2306.04757) | [Model](https://huggingface.co/declare-lab/flan-alpaca-gpt4-xl) | [Leaderboard](https://declare-lab.github.io/instruct-eval/)\n\n<p align=\"center\">\n  <img src=\"https://raw.githubusercontent.com/declare-lab/instruct-eval/main/docs/logo.png\" alt=\"\" width=\"300\" height=\"300\">\n</p>\n\n> 🔥 If you are interested in IQ testing LLMs, check out our new work: [AlgoPuzzleVQA](https://github.com/declare-lab/puzzle-reasoning)\n\n> 📣 Introducing Resta: **Safety Re-alignment of Language Models**. [**Paper**](https://arxiv.org/abs/2402.11746) [**Github**](https://github.com/declare-lab/resta)\n\n> 📣 **Red-Eval**, the benchmark for **Safety** Evaluation of LLMs has been added: [Red-Eval](https://github.com/declare-lab/instruct-eval/tree/main/red-eval)\n\n> 📣 Introducing **Red-Eval** to evaluate the safety of the LLMs using several jailbreaking prompts. With **Red-Eval** one could jailbreak/red-team GPT-4 with a 65.1% attack success rate and ChatGPT could be jailbroken 73% of the time as measured on DangerousQA and HarmfulQA benchmarks. More details are here: [Code](https://github.com/declare-lab/red-instruct) and [Paper](https://arxiv.org/abs/2308.09662).\n\n> 📣 We developed Flacuna by fine-tuning Vicuna-13B on the Flan collection. Flacuna is better than Vicuna at problem-solving. Access the model here [https://huggingface.co/declare-lab/flacuna-13b-v1.0](https://huggingface.co/declare-lab/flacuna-13b-v1.0).\n\n> 📣 The [**InstructEval**](https://declare-lab.net/instruct-eval/) benchmark and leaderboard have been released. \n\n> 📣 The paper reporting Instruction Tuned LLMs on the **InstructEval** benchmark suite has been released on Arxiv. Read it here: [https://arxiv.org/pdf/2306.04757.pdf](https://arxiv.org/pdf/2306.04757.pdf)\n\n> 📣 We are releasing **IMPACT**, a dataset for evaluating the writing capability of LLMs in four aspects: Informative, Professional, Argumentative, and Creative. Download it from Huggingface: [https://huggingface.co/datasets/declare-lab/InstructEvalImpact](https://huggingface.co/datasets/declare-lab/InstructEvalImpact). \n\n> 📣 **FLAN-T5** is also useful in text-to-audio generation. Find our work\nat [https://github.com/declare-lab/tango](https://github.com/declare-lab/tango) if you are interested.\n\nThis repository contains code to evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out\ntasks.\nWe aim to facilitate simple and convenient benchmarking across multiple tasks and models.\n\n### Why?\n\nInstruction-tuned models such as [Flan-T5](https://arxiv.org/abs/2210.11416)\nand [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) represent an exciting direction to approximate the\nperformance of large language models (LLMs) like ChatGPT at lower cost.\nHowever, it is challenging to compare the performance of different models qualitatively.\nTo evaluate how well the models generalize across a wide range of unseen and challenging tasks, we can use academic\nbenchmarks such as [MMLU](https://arxiv.org/abs/2009.03300) and [BBH](https://arxiv.org/abs/2210.09261).\nCompared to existing libraries such as [evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)\nand [HELM](https://github.com/stanford-crfm/helm), this repo enables simple and convenient evaluation for multiple\nmodels.\nNotably, we support most models from HuggingFace Transformers 🤗 (check [here](./docs/models.md) for a list of models we support):\n\n- [AutoModelForCausalLM](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) (\n  eg [GPT-2](https://huggingface.co/gpt2-xl), [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6b)\n  , [OPT-IML](https://huggingface.co/facebook/opt-iml-max-1.3b), [BLOOMZ](https://huggingface.co/bigscience/bloomz-7b1))\n- [AutoModelForSeq2SeqLM](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForSeq2SeqLM) (\n  eg","github_created_at":"2023-03-28T19:06:56+00:00","created_at":"2026-07-11T10:35:08.601178+00:00","updated_at":"2026-07-12T03:16:25.001442+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"benchmarking","name":"benchmarking"},{"slug":"evaluation","name":"evaluation"},{"slug":"instruct-tuning","name":"instruct-tuning"},{"slug":"instruction-following","name":"instruction-following"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"llm","name":"llm"},{"slug":"performance-assessment","name":"performance-assessment"},{"slug":"safety-evaluation","name":"safety-evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":620479896,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:35:09.628Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":853,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":83,"high_count":0,"last_scan_at":"2026-07-11T10:35:10.558Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T03:16:24.922Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-12T03:16:24.922Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-12T03:16:24.922Z"}}}}