Open-LLM-Leaderboard: Open-Style Question Evaluation

We introduce the Open-LLM-Leaderboard to track various LLMs’ performance on open-style questions and reflect their true capability. You can use OSQ-bench questions and prompts to evaluate your models automatically with an LLM-based evaluator. The leaderboard is available for viewing on HuggingFace.

Contents

• Open-LLM-Leaderboard: Open-Style Question Evaluation
  • Contents
  • Pre-Generated Model Answers and Evaluation
  • OSQ-Bench
    • Evaluate a model on OSQ-bench
      • Step 1. Generate model answers to OSQ-bench questions
      • Step 2. Generate GPT-4 evaluation
  • Contributing a model
  • Leaderboards
  • Citation
  • Acknowledgments

Pre-Generated Model Answers and Evaluation

We provide pre-generated model answers and evaluation for models. They can be downloaded using the Huggingface dataset. You can also view them at Google Drive.

import datasets
gpt4_responses = datasets.load_dataset("Open-Style/Open-LLM-Benchmark", "gpt4")

Each data point is represented as the following:

{
  "question": "What is the main function of photosynthetic cells within a plant?",
  "gold_answer": "to convert energy from sunlight into food energy",
  "os_answer": "The main function of photosynthetic cells ...",
  "os_eval": "Correct",
  "mcq_answer": "C",
  "mcq_eval": true,
  "dataset": "ARC"
}

OSQ-Bench

OSQ-bench is a set of questions from datasets MMLU, ARC, WinoGrande, PIQA, CommonsenseQA, Race, MedMCQA, and OpenbookQA that are suitable for open-style answering. To automate the evaluation process, we use LLMs like GPT-4 to act as evaluators and assess the quality of the models' responses.

Evaluate a model on OSQ-bench

Step 1. Generate model answers to OSQ-bench questions

To evaluate a model you need to:

1. Download the benchmark and generate the answers. You can use the Huggingface dataset to download it:

import datasets
import json

eval_set = datasets.load_dataset("Open-Style/Open-LLM-Benchmark", "questions")
grouped_responses = []
for example in eval_set['train']:
    # generate here is a placeholder for your models generations
    response = {"Question": example["question"], "os_answer": generate(example["question"]), "dataset": example["dataset"]}
    dataset = example["dataset"]
    if dataset not in grouped_responses:
        grouped_responses[dataset] = []
    grouped_responses[dataset].append(response)

Or lm-evaluation-harness can be used to generate the answers. To use it first run: pip install lm-eval. Then run the following for the tasks in lm-eval-tasks folder:

lm_eval \
    --model hf \
    --model_args pretrained=[MODEL-NAME] \
    --tasks os_mmlu \
    --device cuda:0 \
    --num_fewshot 0 \
    --include_path ./ \
    --batch_size auto \
    --output_path mmlu.jsonl \
    --log_samples \
    --predict_only 

Step 2. Generate GPT-4 evaluation

In this step, we ask GPT-4 to grade the model's answer by comparing it to the correct answer from the benchmark. For each