BIG-bench

Quick start Colab notebooks

Using SeqIO to inspect and evaluate BIG-bench json tasks:

• load BIG-bench json tasks and inspect examples
• evaluate a t5x model on a BIG-bench json task

Creating new BIG-bench tasks

• lightweight task creation and evaluation
• manually perform BIG-bench tasks (after creating a task in your own branch, use this notebook to manually evaluate and verify that it is behaving correctly)

Quick start instructions to load BIG-bench json tasks using SeqIO :chair:+:telescope:

!pip install git+https://github.com/google/BIG-bench.git # This may take a few minutes

import seqio
from bigbench.bbseqio import tasks

---

## Installation of BIG-bench

**Requirements**

* Python 3.5-3.8
* pytest (for running the automated tests)

**Instructions**

First, clone the repository and set up the environment.
```bash