{"data":{"slug":"vila-lab-open-llm-leaderboard","name":"Open-LLM-Leaderboard","tagline":"Open-LLM-Leaderboard: Open-Style Question Evaluation. Paper at https://arxiv.org/abs/2406.07545","github_url":"https://github.com/VILA-Lab/Open-LLM-Leaderboard","owner":"VILA-Lab","repo":"Open-LLM-Leaderboard","owner_avatar_url":"https://avatars.githubusercontent.com/u/101944607?v=4","primary_language":"Python","stars":53,"forks":7,"topics":["leaderboard","llm-evaluation","llm-leaderboard","llms","open-ended-evaluation","open-ended-question-marker"],"archived":false,"github_pushed_at":"2024-06-27T11:03:32+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/vila-lab-open-llm-leaderboard","markdown_url":"https://www.graphcanon.com/tools/vila-lab-open-llm-leaderboard.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/vila-lab-open-llm-leaderboard","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=vila-lab-open-llm-leaderboard","description":"Open-LLM-Leaderboard: Open-Style Question Evaluation. Paper at https://arxiv.org/abs/2406.07545","homepage_url":"https://huggingface.co/spaces/Open-Style/OSQ-Leaderboard","license":"CC-BY-4.0","open_issues":1,"watchers":2,"ai_summary":null,"readme_excerpt":"<p align=\"center\" width=\"100%\">\n<img src=\"assets/logo.jpg\" alt=\"OSQ\" style=\"width: 28%; min-width: 150px; display: block; margin: auto;\">\n</p>\n\n# Open-LLM-Leaderboard: Open-Style Question Evaluation\n\n\n\n<a href=\"https://arxiv.org/abs/2406.07545\"><img src=\"https://img.shields.io/badge/arXiv-2406.07545-b31b1b.svg\" alt=\"arXiv\"></a>\n\n\n\n\nWe introduce the [Open-LLM-Leaderboard](https://arxiv.org/abs/2406.07545) to track various LLMs’ performance on open-style questions and reflect their true capability.\nYou can use OSQ-bench questions and prompts to evaluate your models automatically with an LLM-based evaluator.\nThe leaderboard is available for viewing on [HuggingFace](https://huggingface.co/spaces/Open-Style/OSQ-Leaderboard). \n\n## Contents\n- [Open-LLM-Leaderboard: Open-Style Question Evaluation](#open-llm-leaderboard-open-style-question-evaluation)\n  - [Contents](#contents)\n  - [Pre-Generated Model Answers and Evaluation](#pre-generated-model-answers-and-evaluation)\n  - [OSQ-Bench](#osq-bench)\n    - [Evaluate a model on OSQ-bench](#evaluate-a-model-on-osq-bench)\n      - [Step 1. Generate model answers to OSQ-bench questions](#step-1-generate-model-answers-to-osq-bench-questions)\n      - [Step 2. Generate GPT-4 evaluation](#step-2-generate-gpt-4-evaluation)\n  - [Contributing a model](#contributing-a-model)\n  - [Leaderboards](#leaderboards)\n  - [Citation](#citation)\n  - [Acknowledgments](#acknowledgments)\n\n## Pre-Generated Model Answers and Evaluation\nWe provide pre-generated model answers and evaluation for models. \nThey can be downloaded using the [Huggingface dataset](https://huggingface.co/datasets/Open-Style/Open-LLM-Benchmark).\nYou can also view them at [Google Drive](https://drive.google.com/drive/folders/1onO68rKkpKGxO8xXSZqPdMJUq3Mbqkaq?usp=sharing).\n```python\nimport datasets\ngpt4_responses = datasets.load_dataset(\"Open-Style/Open-LLM-Benchmark\", \"gpt4\")\n```\nEach data point is represented as the following:\n```json\n{\n  \"question\": \"What is the main function of photosynthetic cells within a plant?\",\n  \"gold_answer\": \"to convert energy from sunlight into food energy\",\n  \"os_answer\": \"The main function of photosynthetic cells ...\",\n  \"os_eval\": \"Correct\",\n  \"mcq_answer\": \"C\",\n  \"mcq_eval\": true,\n  \"dataset\": \"ARC\"\n}\n```\n## OSQ-Bench\nOSQ-bench is a set of questions from datasets MMLU, ARC, WinoGrande, PIQA, CommonsenseQA, Race, MedMCQA, and OpenbookQA that are suitable for open-style answering.\nTo automate the evaluation process, we use LLMs like GPT-4 to act as evaluators and assess the quality of the models' responses.\n\n<div align=center>\n<img width=\"580\" src=\"assets/data_distribution.png\"/>\n</div>\n\n### Evaluate a model on OSQ-bench\n\n#### Step 1. Generate model answers to OSQ-bench questions\nTo evaluate a model you need to:\n\n1. Download the benchmark and generate the answers. You can use the Huggingface dataset to download it:\n```python\nimport datasets\nimport json\n\neval_set = datasets.load_dataset(\"Open-Style/Open-LLM-Benchmark\", \"questions\")\ngrouped_responses = []\nfor example in eval_set['train']:\n    # generate here is a placeholder for your models generations\n    response = {\"Question\": example[\"question\"], \"os_answer\": generate(example[\"question\"]), \"dataset\": example[\"dataset\"]}\n    dataset = example[\"dataset\"]\n    if dataset not in grouped_responses:\n        grouped_responses[dataset] = []\n    grouped_responses[dataset].append(response)\n```\nOr lm-evaluation-harness can be used to generate the answers. To use it first run: `pip install lm-eval`. Then run the following for the tasks in `lm-eval-tasks` folder:\n```\nlm_eval \\\n    --model hf \\\n    --model_args pretrained=[MODEL-NAME] \\\n    --tasks os_mmlu \\\n    --device cuda:0 \\\n    --num_fewshot 0 \\\n    --include_path ./ \\\n    --batch_size auto \\\n    --output_path mmlu.jsonl \\\n    --log_samples \\\n    --predict_only \n```\n#### Step 2. Generate GPT-4 evaluation\nIn this step, we ask GPT-4 to grade the model's answer by comparing it to the correct answer from the benchmark.\nFor each","github_created_at":"2024-06-11T16:31:46+00:00","created_at":"2026-07-15T10:40:32.872795+00:00","updated_at":"2026-07-15T10:40:35.763585+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"leaderboard","name":"leaderboard"},{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"llm-leaderboard","name":"llm-leaderboard"},{"slug":"llms","name":"llms"},{"slug":"open-ended-evaluation","name":"open-ended-evaluation"},{"slug":"open-ended-question-marker","name":"open-ended-question-marker"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":813736154,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-15T10:40:34.024Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":747,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T10:40:34.481Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-15T10:40:33.785Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-15T10:40:33.785Z"},"license_spdx":{"value":"CC-BY-4.0","source":"github.license","observed_at":"2026-07-15T10:40:33.785Z"}}}}