GraphCanon updated 2w · GitHub synced 2w
Decision brief
Key facts about instruct-eval
Good fit when
- When you need to quantitatively evaluate the performance of instruction-tuned large language models such as Alpaca and Flan-T5 on held-out tasks.
- If you require a comprehensive toolset for both benchmarking and safety assessment, including specialized benchmarks like IMPACT and Red-Eval.
Avoid when
- When primarily interested in general model evaluation without a focus on instruction-tuned LMs.
- If your primary interest lies in qualitative assessment rather than quantitative metrics.
- Requirements:
- Min 8 GB RAM; Requires Python environment setup and specific dependencies as outlined in the repository's documentation.
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (879d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- 83 low (83 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install instruct-evalSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A toolset for evaluating the performance of instruction-tuned large language models, including benchmarking and safety assessment.
Capability facts
- Languages
- python
Source: github.language · Aug 7, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 7, 2026)
-Eval** one could jailbreak/red-team GPT-4 with a 65.1% attack success rate and ChatGPT could be jailbroken 73% of the time as measured on DangerousQA and HarmfulQA beSource link
Tags
README
:camel: 🍮 📚 InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Paper | Model | Leaderboard
🔥 If you are interested in IQ testing LLMs, check out our new work: AlgoPuzzleVQA
📣 Introducing Resta: Safety Re-alignment of Language Models. Paper Github
📣 Red-Eval, the benchmark for Safety Evaluation of LLMs has been added: Red-Eval
📣 Introducing Red-Eval to evaluate the safety of the LLMs using several jailbreaking prompts. With Red-Eval one could jailbreak/red-team GPT-4 with a 65.1% attack success rate and ChatGPT could be jailbroken 73% of the time as measured on DangerousQA and HarmfulQA benchmarks. More details are here: Code and Paper.
📣 We developed Flacuna by fine-tuning Vicuna-13B on the Flan collection. Flacuna is better than Vicuna at problem-solving. Access the model here https://huggingface.co/declare-lab/flacuna-13b-v1.0.
📣 The InstructEval benchmark and leaderboard have been released.
📣 The paper reporting Instruction Tuned LLMs on the InstructEval benchmark suite has been released on Arxiv. Read it here: https://arxiv.org/pdf/2306.04757.pdf
📣 We are releasing IMPACT, a dataset for evaluating the writing capability of LLMs in four aspects: Informative, Professional, Argumentative, and Creative. Download it from Huggingface: https://huggingface.co/datasets/declare-lab/InstructEvalImpact.
📣 FLAN-T5 is also useful in text-to-audio generation. Find our work at https://github.com/declare-lab/tango if you are interested.
This repository contains code to evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks. We aim to facilitate simple and convenient benchmarking across multiple tasks and models.
Why?
Instruction-tuned models such as Flan-T5 and Alpaca represent an exciting direction to approximate the performance of large language models (LLMs) like ChatGPT at lower cost. However, it is challenging to compare the performance of different models qualitatively. To evaluate how well the models generalize across a wide range of unseen and challenging tasks, we can use academic benchmarks such as MMLU and BBH. Compared to existing libraries such as evaluation-harness and HELM, this repo enables simple and convenient evaluation for multiple models. Notably, we support most models from HuggingFace Transformers 🤗 (check here for a list of models we support):
- AutoModelForCausalLM ( eg GPT-2, GPT-J , OPT-IML, BLOOMZ)
- AutoModelForSeq2SeqLM ( eg
For agents
This page has a .md twin and JSON over the API.