--- title: "model_card" type: "tool" slug: "bigscience-workshop-model-card" canonical_url: "https://www.graphcanon.com/tools/bigscience-workshop-model-card" github_url: "https://github.com/bigscience-workshop/model_card" homepage_url: null stars: 26 forks: 5 primary_language: null license: "Apache-2.0" archived: false categories: ["llm-frameworks", "model-training", "vector-databases"] tags: [] updated_at: "2026-07-11T23:08:47.130857+00:00" --- # model_card > model_card ## Facts - Repository: https://github.com/bigscience-workshop/model_card - Stars: 26 · Forks: 5 · Open issues: 0 · Watchers: 2 - License: Apache-2.0 - Last pushed: 2022-07-11T18:57:23+00:00 ## Trust & health _Signals computed from public GitHub metadata. Not a security guarantee._ - Maintenance: Dormant (computed 2026-07-11T23:08:32.511Z) - Security scan: No lockfile (0 critical, 0 high, 0 medium, 0 low) · last scan 2026-07-11T23:08:32.844Z - Full report: [trust report](/tools/bigscience-workshop-model-card/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/bigscience-workshop-model-card/trust) ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) - [Model Training](/categories/model-training.md) - [Vector Databases](/categories/vector-databases.md) ## Category neighbours (exploratory) _Same-category tools for discovery only - not curated alternatives. Cap shown at six._ - [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active] - [tensorflow](/tools/tensorflow-tensorflow.md) - An Open Source Machine Learning Framework for Everyone (★ 196,300) [Very active] - [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active] - [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active] - [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active] - [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 162,482) [Very active] _+ 2 more not listed._ ## README (excerpt) _Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._ ```text --- license: bigscience-bloom-rail-1.0 language: - ak - ar - as - bm - bn - ca - code - en - es - eu - fon - fr - gu - hi - id - ig - ki - kn - lg - ln - ml - mr - ne - nso - ny - or - pa - pt - rn - rw - sn - st - sw - ta - te - tn - ts - tum - tw - ur - vi - wo - xh - yo - zh - zhs - zht - zu --- # DEPRECATED. This is now live at https://huggingface.co/bigscience/bloom . Please make additional changes there! #

BLOOM LM
_BigScience Large Open-science Open-access Multilingual Language Model_
Model Card

BigScience Logo Version 1.0 / 25.May.2022 ## Table of Contents 1. [Model Details](#model-details) 2. [Uses](#uses) 3. [Training Data](#training-data) 4. [Risks and Limitations](#risks-and-limitations) 5. [Evaluation](#evaluation) 6. [Recommendations](#recommendations) 7. [Glossary and Calculations](#glossary-and-calculations) 8. [More Information](#more-information) 9. [Model Card Authors](#model-card-authors) ## Model Details ### Basics *This section provides information for anyone who wants to know about the model.*
Click to expand
**Developed by:** BigScience ([website](https://bigscience.huggingface.co)) * All collaborators are either volunteers or have an agreement with their employer. *(Further breakdown of participants forthcoming.)* **Model Type:** Transformer-based Language Model **Version:** 1.0.0 **Languages:** Multiple; see [training data](#training-data) **License:** RAIL License v1.0 ([link](https://huggingface.co/spaces/bigscience/license)) **Release Date Estimate:** Monday, 11.July.2022 **Send Questions to:** bigscience-contact@googlegroups.com **Cite as:** BigScience, _BigScience Language Open-source Open-access Multilingual (BLOOM) Language Model_. International, May 2021-May 2022 **Funded by:** * The French government. * Hugging Face ([website](https://huggingface.co)). * Organizations of contributors. *(Further breakdown of organizations forthcoming.)*
### Technical Specifications *This section provides information for people who work on model development.*
Click to expand
Please see [the BLOOM training README](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml#readme) for full details on replicating training. **Model Architecture:** Modified from Megatron-LM GPT2 (see [paper](https://arxiv.org/abs/1909.08053), [BLOOM Megatron code](https://github.com/bigscience-workshop/Megatron-DeepSpeed)): * Decoder-only architecture * Layer normalization applied to word embeddings layer (`StableEmbedding`; see [code](https://github.com/facebookresearch/bitsandbytes), [paper](https://arxiv.org/pdf/2110.02861.pdf)) * ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions * 176 billion parameters: * 70 layers, 112 attention heads * Hidden layers are 14336-dimensional * Sequence length of 2048 tokens used (see [BLOOM tokenizer](https://huggingface.co/bigscience/tokenizer), [tokenizer description](#tokenization)) **Objective Function:** Cross Entropy with mean reduction (see [API documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss)). **Compute infrastructure:** Jean Zay Public Supercomputer, provided by the French government (see [announcement](https://www.enseignementsup-recherche.gouv.fr/fr/signature-du-marche-d-acquisition-de-l-un-des-supercalculateurs-les-plus-puissants-d-europe-46733)). * Hardware: 384 A100 80GB GPUs (48 nodes): * Additional 32 A100 80GB GPUs (4 nodes) in reserve * 8 GPUs per node Using NVLink 4 inter-gpu connects, 4 OmniPath links * CPU: AMD * CPU memory: 512GB per node * GPU memory: 640GB per node * Inter-node connec ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/bigscience-workshop-model-card`](/api/graphcanon/tools/bigscience-workshop-model-card) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_