--- title: "petals" type: "tool" slug: "bigscience-workshop-petals" canonical_url: "https://www.graphcanon.com/tools/bigscience-workshop-petals" github_url: "https://github.com/bigscience-workshop/petals" homepage_url: "https://petals.dev" stars: 10266 forks: 621 primary_language: "Python" license: "MIT" categories: ["llm-frameworks"] tags: ["deep-learning", "falcon", "language-models", "gpt", "guanaco", "bloom", "chatbot", "distributed-systems"] updated_at: "2026-07-07T18:33:51.2373+00:00" --- # petals > Run large language models at home, BitTorrent-style. Petals enables running large language models locally with speed-ups for fine-tuning and inference. Supports distributed execution of Llama, Mixtral, Falcon, and BLOOM with up to 405B parameters. ## Facts - Repository: https://github.com/bigscience-workshop/petals - Homepage: https://petals.dev - Stars: 10,266 · Forks: 621 · Open issues: 112 · Watchers: 106 - Primary language: Python - License: MIT - Last pushed: 2024-09-07T11:54:28+00:00 ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) ## Tags deep-learning, falcon, language-models, gpt, guanaco, bloom, chatbot, distributed-systems ## Related tools - [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,659) - [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019) - [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347) - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575) - [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702) - [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,711) - [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,610) - [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,150) ## README (excerpt) ```text


Run large language models at home, BitTorrent-style.
Fine-tuning and inference up to 10x faster than offloading


Generate text with distributed **Llama 3.1** (up to 405B), **Mixtral** (8x22B), **Falcon** (40B+) or **BLOOM** (176B) and fine‑tune them for your own tasks — right from your desktop computer or Google Colab: ```python from transformers import AutoTokenizer from petals import AutoDistributedModelForCausalLM # Choose any model available at https://health.petals.dev model_name = "meta-llama/Meta-Llama-3.1-405B-Instruct" # Connect to a distributed network hosting model layers tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoDistributedModelForCausalLM.from_pretrained(model_name) # Run the model as if it were on your computer inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"] outputs = model.generate(inputs, max_new_tokens=5) print(tokenizer.decode(outputs[0])) # A cat sat on a mat... ```

🚀  Try now in Colab

🦙 **Want to run Llama?** [Request access](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct) to its weights, then run `huggingface-cli login` in the terminal before loading the model. Or just try it in our [chatbot app](https://chat.petals.dev). 🔏 **Privacy.** Your data will be processed with the help of other people in the public swarm. Learn more about privacy [here](https://github.com/bigscience-workshop/petals/wiki/Security,-privacy,-and-AI-safety). For sensitive data, you can set up a [private swarm](https://github.com/bigscience-workshop/petals/wiki/Launch-your-own-swarm) among people you trust. 💬 **Any questions?** Ping us in [our Discord](https://discord.gg/KdThf2bWVU)! ## Connect your GPU and increase Petals capacity Petals is a community-run system — we rely on people sharing their GPUs. You can help serving one of the [available models](https://health.petals.dev) or host a new model from 🤗 [Model Hub](https://huggingface.co/models)! As an example, here is how to host a part of [Llama 3.1 (405B) Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct) on your GPU: 🦙 **Want to host Llama?** [Request access](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct) to its weights, then run `huggingface-cli login` in the terminal before loading the model. 🐧 **Linux + Anaconda.** Run these commands for NVIDIA GPUs (or follow [this](https://github.com/bigscience-workshop/petals/wiki/Running-on-AMD-GPU) for AMD): ```bash conda install pytorch pytorch-cuda=11.7 -c pytorch -c nvidia pip install git+https://github.com/bigscience-workshop/petals python -m petals.cli.run_server meta-llama/Meta-Llama-3.1-405B-Instruct ``` 🪟 **Windows + WSL.** Follow [this guide](https://github.com/bigscience-workshop/petals/wiki/Run-Petals-server-on-Windows) on our Wiki. 🐋 **Docker.** Run our [Docker](https://www.docker.com) image for NVIDIA GPUs (or follow [this](https://github.com/bigscience-workshop/petals/wiki/Running-on-AMD-GPU) for AMD): ```bash sudo docker run -p 31330:31330 --ipc host --gpus all --volume petals-cache:/cache --rm \ learningathome/petals:main \ python -m petals.cli.run_server --port 31330 meta-llama/Meta-Llama-3.1-405B-Instruct ``` 🍏 **macOS + Apple M1/M2 GPU.** Install [Homebrew](https://brew.sh/), then run these commands: ```bash brew install python python3 -m pip install git+https://github.com/bigscience ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/bigscience-workshop-petals`](/api/graphcanon/tools/bigscience-workshop-petals) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_