{"data":{"slug":"agi-edgerunners-llm-adapters","name":"LLM-Adapters","tagline":"Code for EMNLP 2023 Paper on Parameter-Efficient Fine-Tuning of LLMs","github_url":"https://github.com/AGI-Edgerunners/LLM-Adapters","owner":"AGI-Edgerunners","repo":"LLM-Adapters","owner_avatar_url":"https://avatars.githubusercontent.com/u/128879899?v=4","primary_language":"Python","stars":1235,"forks":116,"topics":["adapters","fine-tuning","large-language-models","parameter-efficient"],"archived":false,"github_pushed_at":"2024-03-10T08:20:15+00:00","maintenance_label":"Dormant","stars_delta_30d":1,"url":"https://www.graphcanon.com/tools/agi-edgerunners-llm-adapters","markdown_url":"https://www.graphcanon.com/tools/agi-edgerunners-llm-adapters.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/agi-edgerunners-llm-adapters","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=agi-edgerunners-llm-adapters","description":"Code for our EMNLP 2023 Paper: \"LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models\"","homepage_url":"https://arxiv.org/abs/2304.01933","license":"Apache-2.0","open_issues":55,"watchers":10,"ai_summary":"Repository contains code and resources related to the research paper on developing an adapter family for parameter-efficient fine-tuning of large language models.","readme_excerpt":"

\n\n

LLM-Adapters

\n

\n\n

\n

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

\n

\nLLM-Adapters is an easy-to-use framework that integrates various adapters into LLMs and can execute adapter-based PEFT methods of LLMs for different tasks. LLM-Adapter is an extension of HuggingFace's PEFT library, many thanks for their amazing work! Please find our paper at this link: https://arxiv.org/abs/2304.01933.\n\nThe framework includes state-of-the-art open-access LLMs: LLaMa, OPT, BLOOM, and GPT-J, as well as widely used adapters such as Bottleneck adapters, Parallel adapters, and LoRA.\n\nSupported Adapters:\n\n1. LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/pdf/2106.09685.pdf)\n2. AdapterH: [Parameter-Efficient Transfer Learning for NLP](https://arxiv.org/pdf/1902.00751.pdf)\n3. AdapterP: [GMAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://arxiv.org/pdf/2005.00052.pdf)\n4. Parallel: [TOWARDS A UNIFIED VIEW OF PARAMETER-EFFICIENT TRANSFER LEARNING](https://arxiv.org/pdf/2110.04366.pdf)\n5. Prefix Tuning: [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://aclanthology.org/2021.acl-long.353/), [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://arxiv.org/pdf/2110.07602.pdf)\n6. P-Tuning: [GPT Understands, Too](https://arxiv.org/pdf/2103.10385.pdf)\n7. Prompt Tuning: [The Power of Scale for Parameter-Efficient Prompt Tuning](https://arxiv.org/pdf/2104.08691.pdf) \n\n## Latest News 🔥🔥\n\n* [2023-08-10] LLM-Adapters has been accepted by EMNLP 2023.\n* [2023-07-16] we released commonsense170k dataset and the The LLaMA-13B-Parallel model outformances ChatGPT on 8 commonsense benchmarks.\n* [2023-04-21] We released math10k dataset and the [LLaMA-13B adapter checkpoints](https://drive.google.com/file/d/1NqUv-Hn_mAkGXsUOqpJKmPKW5Gp8mRlO/view?usp=sharing). The LLaMA-13B-Parallel model achieves **91%** of GPT-3.5 performance!\n* [2023-04-10] We can support GPT-Neo and ChatGLM now!\n* [2023-04-04] [Release code and dataset](https://github.com/AGI-Edgerunners/LLM-Adapters)\n\n## Special Announcement\nThe `math_10k.json` data is collected with the training sets of GSM8K, MAWPS, and AQuA(1000 examples). However, MAWPS consists of AddSub, MultiArith, SingleOp, SingleEq, SimulEq-S, SimulEq-L. Thus, we can't utilize MultiArith, AddSub, and SingleEq as evaluation benchmarks with models trained with `math_10k.json`. We evaluate the PEFT methods on the MAWPS test set instead, and the result table has been updated (The findings in the paper are consistent). Furthermore, two variations of `math_10k.json` have been uploaded, `math_7K.json` where the MAWPS samples have been deleted, and `math_14k.json` where the MAWPS samples have been deleted as well and we combine ChatGPT and GPT-4 rationales. Sincerely apologize for any inconvenience!\n\n## Setup\n\n1. Install dependencies\n```bash\npip install -r requirements.txt\n```\n\n2. Set environment variables, or modify the files referencing `BASE_MODEL`:\n\n```bash\n# Files referencing `BASE_MODEL`\n# export_hf_checkpoint.py\n# export_state_dict_checkpoint.py\n\nexport BASE_MODEL=yahma/llama-7b-hf\n```\n\nBoth `finetune.py` and `generate.py` use `--base_model` flag as shown further below.\n\n3. If bitsandbytes doesn't work, [install it from source.](https://github.com/TimDettmers/bitsandbytes/blob/main/compile_from_source.md) Windows users can follow [these instructions](https://github.com/tloen/alpaca-lora/issues/17).\n\n## Training(finetune.py)\n\nThis file contains some code related to prompt construction and tokenization.In this file, specify different adapters and different sets of data, so that different models can be trained. \n\nExample usage for multiple GPUs:\n\n```bash\nWORLD_SIZE=2 CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 --master_port=3192 finetune.py \\\n --base_m","github_created_at":"2023-03-29T13:34:08+00:00","created_at":"2026-07-11T11:39:21.735675+00:00","updated_at":"2026-09-20T06:29:58.864873+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"adapters","name":"adapters"},{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"parameter-efficient","name":"parameter-efficient"}],"trust":{"provenance":{"is_fork":false,"github_id":620843043,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-19T18:25:16.444Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":923,"last_release_at":null,"stars_delta_30d":1,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:39:23.047Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-19T18:25:17.496Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-19T18:25:17.496Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-19T18:25:17.496Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Optimizing resource usage when you need to fine-tune large language models without altering their core parameters","Your project involves parameter-efficient updates, following the latest research trends outlined in EMNLP 2023"],"when_not_to_use":["You require a full retraining approach that modifies all model weights, not just adapters","Your project timeline does not allow for integrating and testing new methodologies from recent papers like EMNLP 2023"],"source":"enrich:decision_facts","observed_at":"2026-07-12T12:09:26.787Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"LLM-Adapters offers Python-based tools for efficient fine-tuning of language models with Apache-2.0 licensing."}]}}