{"data":{"slug":"google-big-bench","name":"BIG-bench","tagline":"Collaborative benchmark for language model capabilities","github_url":"https://github.com/google/BIG-bench","owner":"google","repo":"BIG-bench","owner_avatar_url":"https://avatars.githubusercontent.com/u/1342004?v=4","primary_language":"Python","stars":3248,"forks":615,"topics":[],"archived":true,"github_pushed_at":"2024-07-19T11:57:37+00:00","maintenance_label":"Archived","url":"https://www.graphcanon.com/tools/google-big-bench","markdown_url":"https://www.graphcanon.com/tools/google-big-bench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/google-big-bench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=google-big-bench","description":"Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models","homepage_url":null,"license":"Apache-2.0","open_issues":106,"watchers":45,"ai_summary":"Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models through various tasks and evaluation methods.","readme_excerpt":"## Quick start Colab notebooks\n\n**Using [SeqIO](https://github.com/google/seqio) to inspect and evaluate BIG-bench json tasks**:\n- [load BIG-bench json tasks and inspect examples](https://colab.research.google.com/github/google/BIG-bench/blob/main/bigbench/bbseqio/docs/quick_start.ipynb) \n\"Open\n- [evaluate a t5x model on a BIG-bench json task](https://colab.research.google.com/github/google/BIG-bench/blob/main/bigbench/bbseqio/docs/t5x_eval.ipynb) \"Open\n\n**Creating new BIG-bench tasks**\n\n- [lightweight task creation and evaluation](https://colab.research.google.com/github/google/BIG-bench/blob/main/notebooks/colab_examples.ipynb) \"Open\n- [manually perform BIG-bench tasks](https://colab.research.google.com/github/google/BIG-bench/blob/main/notebooks/TaskTestingNotebook.ipynb) \"Open (after creating a task in your own branch, use this notebook to manually evaluate and verify that it is behaving correctly)\n\n---\n\n## Quick start instructions to load BIG-bench json tasks using [SeqIO](https://github.com/google/seqio) :chair:+:telescope:\n\n```\n!pip install git+https://github.com/google/BIG-bench.git # This may take a few minutes\n\nimport seqio\nfrom bigbench.bbseqio import tasks\n\n---\n\n## Installation of BIG-bench\n\n**Requirements**\n\n* Python 3.5-3.8\n* pytest (for running the automated tests)\n\n**Instructions**\n\nFirst, clone the repository and set up the environment.\n```bash","github_created_at":"2021-01-15T23:28:20+00:00","created_at":"2026-07-11T10:30:26.2089+00:00","updated_at":"2026-07-12T06:01:08.371719+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"tasks-creation","name":"tasks creation"},{"slug":"evaluation","name":"evaluation"},{"slug":"seqio","name":"seqio"},{"slug":"language-models","name":"language-models"},{"slug":"t5x","name":"t5x"},{"slug":"benchmarking","name":"benchmarking"}],"trust":{"provenance":{"is_fork":false,"github_id":330044683,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-12T06:01:02.160Z","maintenance":{"label":"Archived","score":8,"methodology":"github_public_v1","releases_90d":0,"days_since_push":722,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":324,"high_count":0,"last_scan_at":"2026-07-11T10:30:29.723Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T06:01:08.273Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-12T06:01:08.273Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-12T06:01:08.273Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Python 3.5-3.8 required.","`pytest` is necessary for running automated tests."],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"requires_docker":false},"when_to_use":["When you need a comprehensive benchmark that evaluates language models across various tasks and includes methods for extrapolating model capabilities.","When your project requires tools specifically designed to measure human-like understanding of AI models beyond mere imitation."],"when_not_to_use":["If you are looking for a tool that simplifies benchmarking with minimal configuration, BIG-bench requires setting up an environment and can be more complex compared to streamlined benchmark tools.","As BIG-bench relies on collaboration across various tasks and contributions from the community, it might not be ideal if you need benchmark tasks or evaluations immediately available without potential","If your project does not require advanced extrapolation techniques for measuring model capabilities over a wide range of benchmarks, simpler evaluation tools may suffice."],"source":"enrich:decision_facts","observed_at":"2026-07-11T11:13:30.403Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Python 3.5-3.8 required.; `pytest` is necessary for running automated tests."},{"label":"Adopt for","value":"Decision-critical facts for BIG-bench"}]}}