LLM-Adapters

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

LLM-Adapters is an easy-to-use framework that integrates various adapters into LLMs and can execute adapter-based PEFT methods of LLMs for different tasks. LLM-Adapter is an extension of HuggingFace's PEFT library, many thanks for their amazing work! Please find our paper at this link: https://arxiv.org/abs/2304.01933.

The framework includes state-of-the-art open-access LLMs: LLaMa, OPT, BLOOM, and GPT-J, as well as widely used adapters such as Bottleneck adapters, Parallel adapters, and LoRA.

Supported Adapters:

1. LoRA: LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
2. AdapterH: Parameter-Efficient Transfer Learning for NLP
3. AdapterP: GMAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
4. Parallel: TOWARDS A UNIFIED VIEW OF PARAMETER-EFFICIENT TRANSFER LEARNING
5. Prefix Tuning: Prefix-Tuning: Optimizing Continuous Prompts for Generation, P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
6. P-Tuning: GPT Understands, Too
7. Prompt Tuning: The Power of Scale for Parameter-Efficient Prompt Tuning

Latest News 🔥🔥

• [2023-08-10] LLM-Adapters has been accepted by EMNLP 2023.
• [2023-07-16] we released commonsense170k dataset and the The LLaMA-13B-Parallel model outformances ChatGPT on 8 commonsense benchmarks.
• [2023-04-21] We released math10k dataset and the LLaMA-13B adapter checkpoints. The LLaMA-13B-Parallel model achieves 91% of GPT-3.5 performance!
• [2023-04-10] We can support GPT-Neo and ChatGLM now!
• [2023-04-04] Release code and dataset

Special Announcement

The math_10k.json data is collected with the training sets of GSM8K, MAWPS, and AQuA(1000 examples). However, MAWPS consists of AddSub, MultiArith, SingleOp, SingleEq, SimulEq-S, SimulEq-L. Thus, we can't utilize MultiArith, AddSub, and SingleEq as evaluation benchmarks with models trained with math_10k.json. We evaluate the PEFT methods on the MAWPS test set instead, and the result table has been updated (The findings in the paper are consistent). Furthermore, two variations of math_10k.json have been uploaded, math_7K.json where the MAWPS samples have been deleted, and math_14k.json where the MAWPS samples have been deleted as well and we combine ChatGPT and GPT-4 rationales. Sincerely apologize for any inconvenience!

Setup

1. Install dependencies

pip install -r requirements.txt

2. Set environment variables, or modify the files referencing BASE_MODEL:

# Files referencing `BASE_MODEL`
# export_hf_checkpoint.py
# export_state_dict_checkpoint.py

export BASE_MODEL=yahma/llama-7b-hf

Both finetune.py and generate.py use --base_model flag as shown further below.

3. If bitsandbytes doesn't work, install it from source. Windows users can follow these instructions.

Training(finetune.py)

This file contains some code related to prompt construction and tokenization.In this file, specify different adapters and different sets of data, so that different models can be trained.

Example usage for multiple GPUs:

WORLD_SIZE=2 CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 --master_port=3192 finetune.py \
  --base_m