Installation

First, we have to install all the libraries listed in requirements.txt

pip install -r requirements.txt

Step by step installation with conda

Create a new conda environment and activate it

conda create -n env
conda activate env

Install the pytorch version compatible with your version of cuda here, for example the following command works with cuda 11.6

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia

Install transformers and peft

conda install -c huggingface transformers 
pip install git+https://github.com/huggingface/peft.git

Note that you can install the latest stable version of transformers by using

pip install git+https://github.com/huggingface/transformers

Install datasets, accelerate and huggingface_hub

conda install -c huggingface -c conda-forge datasets
conda install -c conda-forge accelerate
conda install -c conda-forge huggingface_hub

Finally, install bitsandbytes and wandb

pip install bitsandbytes
pip install wandb

To get the full list of arguments with descriptions you can run the following command on any script:

python scripts/some_script.py --help

Before you run any of the scripts make sure you are logged in and can push to the hub:

huggingface-cli login

Make sure you are logged in wandb:

wandb login

Now that everything is done, you can clone the repository and get into the corresponding directory.

Inference hardware requirements

In FP32 the model requires more than 60GB of RAM, you can load it in FP16 or BF16 in ~30GB, or in 8bit under 20GB of RAM with