{"data":{"slug":"declare-lab-instruct-eval","name":"instruct-eval","tagline":"Quantitative evaluation for instruction-tuned language models","github_url":"https://github.com/declare-lab/instruct-eval","owner":"declare-lab","repo":"instruct-eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/59164695?v=4","primary_language":"Python","stars":552,"forks":45,"topics":["instruct-tuning","llm"],"archived":false,"github_pushed_at":"2024-03-10T05:00:00+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/declare-lab-instruct-eval","markdown_url":"https://www.graphcanon.com/tools/declare-lab-instruct-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/declare-lab-instruct-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=declare-lab-instruct-eval","description":"This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. ","homepage_url":"https://declare-lab.github.io/instruct-eval/","license":"Apache-2.0","open_issues":24,"watchers":9,"ai_summary":"A toolset for evaluating the performance of instruction-tuned large language models, including benchmarking and safety assessment.","readme_excerpt":"## :camel: 🍮 📚 InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models\n\n[Paper](https://arxiv.org/abs/2306.04757) | [Model](https://huggingface.co/declare-lab/flan-alpaca-gpt4-xl) | [Leaderboard](https://declare-lab.github.io/instruct-eval/)\n\n<p align=\"center\">\n  <img src=\"https://raw.githubusercontent.com/declare-lab/instruct-eval/main/docs/logo.png\" alt=\"\" width=\"300\" height=\"300\">\n</p>\n\n> 🔥 If you are interested in IQ testing LLMs, check out our new work: [AlgoPuzzleVQA](https://github.com/declare-lab/puzzle-reasoning)\n\n> 📣 Introducing Resta: **Safety Re-alignment of Language Models**. [**Paper**](https://arxiv.org/abs/2402.11746) [**Github**](https://github.com/declare-lab/resta)\n\n> 📣 **Red-Eval**, the benchmark for **Safety** Evaluation of LLMs has been added: [Red-Eval](https://github.com/declare-lab/instruct-eval/tree/main/red-eval)\n\n> 📣 Introducing **Red-Eval** to evaluate the safety of the LLMs using several jailbreaking prompts. With **Red-Eval** one could jailbreak/red-team GPT-4 with a 65.1% attack success rate and ChatGPT could be jailbroken 73% of the time as measured on DangerousQA and HarmfulQA benchmarks. More details are here: [Code](https://github.com/declare-lab/red-instruct) and [Paper](https://arxiv.org/abs/2308.09662).\n\n> 📣 We developed Flacuna by fine-tuning Vicuna-13B on the Flan collection. Flacuna is better than Vicuna at problem-solving. Access the model here [https://huggingface.co/declare-lab/flacuna-13b-v1.0](https://huggingface.co/declare-lab/flacuna-13b-v1.0).\n\n> 📣 The [**InstructEval**](https://declare-lab.net/instruct-eval/) benchmark and leaderboard have been released. \n\n> 📣 The paper reporting Instruction Tuned LLMs on the **InstructEval** benchmark suite has been released on Arxiv. Read it here: [https://arxiv.org/pdf/2306.04757.pdf](https://arxiv.org/pdf/2306.04757.pdf)\n\n> 📣 We are releasing **IMPACT**, a dataset for evaluating the writing capability of LLMs in four aspects: Informative, Professional, Argumentative, and Creative. Download it from Huggingface: [https://huggingface.co/datasets/declare-lab/InstructEvalImpact](https://huggingface.co/datasets/declare-lab/InstructEvalImpact). \n\n> 📣 **FLAN-T5** is also useful in text-to-audio generation. Find our work\nat [https://github.com/declare-lab/tango](https://github.com/declare-lab/tango) if you are interested.\n\nThis repository contains code to evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out\ntasks.\nWe aim to facilitate simple and convenient benchmarking across multiple tasks and models.\n\n### Why?\n\nInstruction-tuned models such as [Flan-T5](https://arxiv.org/abs/2210.11416)\nand [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) represent an exciting direction to approximate the\nperformance of large language models (LLMs) like ChatGPT at lower cost.\nHowever, it is challenging to compare the performance of different models qualitatively.\nTo evaluate how well the models generalize across a wide range of unseen and challenging tasks, we can use academic\nbenchmarks such as [MMLU](https://arxiv.org/abs/2009.03300) and [BBH](https://arxiv.org/abs/2210.09261).\nCompared to existing libraries such as [evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)\nand [HELM](https://github.com/stanford-crfm/helm), this repo enables simple and convenient evaluation for multiple\nmodels.\nNotably, we support most models from HuggingFace Transformers 🤗 (check [here](./docs/models.md) for a list of models we support):\n\n- [AutoModelForCausalLM](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForCausalLM) (\n  eg [GPT-2](https://huggingface.co/gpt2-xl), [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6b)\n  , [OPT-IML](https://huggingface.co/facebook/opt-iml-max-1.3b), [BLOOMZ](https://huggingface.co/bigscience/bloomz-7b1))\n- [AutoModelForSeq2SeqLM](https://huggingface.co/docs/transformers/model_doc/auto#transformers.AutoModelForSeq2SeqLM) (\n  eg","github_created_at":"2023-03-28T19:06:56+00:00","created_at":"2026-07-11T10:35:08.601178+00:00","updated_at":"2026-08-07T00:01:55.170625+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"benchmarking","name":"benchmarking"},{"slug":"evaluation","name":"evaluation"},{"slug":"instruct-tuning","name":"instruct-tuning"},{"slug":"llm","name":"llm"},{"slug":"safety","name":"safety"}],"trust":{"provenance":{"is_fork":false,"github_id":620479896,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-07T00:01:54.454Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":879,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":83,"high_count":0,"last_scan_at":"2026-07-11T10:35:10.558Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-07T00:01:54.891Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-07T00:01:54.891Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-07T00:01:54.891Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires Python environment setup and specific dependencies as outlined in the repository's documentation."],"min_ram_gb":8,"requires_docker":false},"constraints":{"min_ram_gb":8,"requires_docker":false},"when_to_use":["When you need to quantitatively evaluate the performance of instruction-tuned large language models such as Alpaca and Flan-T5 on held-out tasks.","If you require a comprehensive toolset for both benchmarking and safety assessment, including specialized benchmarks like IMPACT and Red-Eval.","For research purposes where simple and convenient benchmarking across multiple task types and models is needed."],"when_not_to_use":["When primarily interested in general model evaluation without a focus on instruction-tuned LMs.","If your primary interest lies in qualitative assessment rather than quantitative metrics.","If you need support for non-HuggingFace Transformer models, as instruct-eval mainly supports models from the HuggingFace ecosystem."],"source":"enrich:decision_facts","observed_at":"2026-07-12T12:50:46.728Z"},"constraint_facets":{"min_ram_gb":8,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Min 8 GB RAM; Requires Python environment setup and specific dependencies as outlined in the repository's documentation."},{"label":"Adopt for","value":"Key facts about instruct-eval"},{"label":"License detail","value":"The tool is distributed under Apache-2.0 license"}]}}