{"data":{"slug":"bigscience-workshop-model-card","name":"model_card","tagline":"model_card","github_url":"https://github.com/bigscience-workshop/model_card","owner":"bigscience-workshop","repo":"model_card","owner_avatar_url":"https://avatars.githubusercontent.com/u/82455566?v=4","primary_language":null,"stars":26,"forks":5,"topics":[],"archived":false,"github_pushed_at":"2022-07-11T18:57:23+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/bigscience-workshop-model-card","markdown_url":"https://www.graphcanon.com/tools/bigscience-workshop-model-card.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bigscience-workshop-model-card","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bigscience-workshop-model-card","description":null,"homepage_url":null,"license":"Apache-2.0","open_issues":0,"watchers":2,"ai_summary":null,"readme_excerpt":"---\nlicense: bigscience-bloom-rail-1.0\nlanguage:\n- ak\n- ar\n- as\n- bm\n- bn\n- ca\n- code\n- en\n- es\n- eu\n- fon\n- fr\n- gu\n- hi\n- id\n- ig\n- ki\n- kn\n- lg\n- ln\n- ml\n- mr\n- ne\n- nso\n- ny\n- or\n- pa\n- pt\n- rn\n- rw\n- sn\n- st\n- sw\n- ta\n- te\n- tn\n- ts\n- tum\n- tw\n- ur\n- vi\n- wo\n- xh\n- yo\n- zh\n- zhs\n- zht\n- zu\n---\n\n# DEPRECATED. This is now live at https://huggingface.co/bigscience/bloom . Please make additional changes there!\n\n# <p>BLOOM LM<br/> _BigScience Large Open-science Open-access Multilingual Language Model_ <br/>Model Card</p>\n<img src=\"https://assets.website-files.com/6139f3cdcbbff3a68486761d/613cd8997b270da063e230c5_Tekengebied%201-p-500.png\" alt=\"BigScience Logo\" width=\"200\"/>\n\n\nVersion 1.0 / 25.May.2022\n\n## Table of Contents\n1. [Model Details](#model-details)\n2. [Uses](#uses)\n3. [Training Data](#training-data)\n4. [Risks and Limitations](#risks-and-limitations)\n5. [Evaluation](#evaluation)\n6. [Recommendations](#recommendations)\n7. [Glossary and Calculations](#glossary-and-calculations)\n8. [More Information](#more-information)\n9. [Model Card Authors](#model-card-authors)\n\n## Model Details  \n\n### Basics\n*This section provides information for anyone who wants to know about the model.*\n\n<details>\n<summary>Click to expand</summary> <br/>\n    \n**Developed by:** BigScience ([website](https://bigscience.huggingface.co))\n\n* All collaborators are either volunteers or have an agreement with their employer. *(Further breakdown of participants forthcoming.)*\n    \n**Model Type:** Transformer-based Language Model\n\n**Version:** 1.0.0\n\n**Languages:** Multiple; see [training data](#training-data)\n\n**License:** RAIL License v1.0 ([link](https://huggingface.co/spaces/bigscience/license))\n\n**Release Date Estimate:** Monday, 11.July.2022\n\n**Send Questions to:** bigscience-contact@googlegroups.com\n\n**Cite as:** BigScience, _BigScience Language Open-source Open-access Multilingual (BLOOM) Language Model_. International, May 2021-May 2022\n\n**Funded by:** \n    \n* The French government.\n\n* Hugging Face ([website](https://huggingface.co)).\n\n* Organizations of contributors.  *(Further breakdown of organizations forthcoming.)*\n\n</details>\n\n### Technical Specifications\n*This section provides information for people who work on model development.*\n\n<details>\n<summary>Click to expand</summary><br/>\n\nPlease see [the BLOOM training README](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml#readme) for full details on replicating training.\n\n**Model Architecture:** Modified from Megatron-LM GPT2 (see [paper](https://arxiv.org/abs/1909.08053), [BLOOM Megatron code](https://github.com/bigscience-workshop/Megatron-DeepSpeed)):\n\n* Decoder-only architecture\n\n* Layer normalization applied to word embeddings layer (`StableEmbedding`; see [code](https://github.com/facebookresearch/bitsandbytes), [paper](https://arxiv.org/pdf/2110.02861.pdf))\n\n* ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions\n\n* 176 billion parameters:\n\n    * 70 layers, 112 attention heads\n\n    * Hidden layers are 14336-dimensional\n\n    * Sequence length of 2048 tokens used (see [BLOOM tokenizer](https://huggingface.co/bigscience/tokenizer), [tokenizer description](#tokenization))\n\n**Objective Function:** Cross Entropy with mean reduction (see [API documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss)).\n    \n**Compute infrastructure:** Jean Zay Public Supercomputer, provided by the French government (see [announcement](https://www.enseignementsup-recherche.gouv.fr/fr/signature-du-marche-d-acquisition-de-l-un-des-supercalculateurs-les-plus-puissants-d-europe-46733)).\n\n* Hardware: 384 A100 80GB GPUs (48 nodes):\n    \n    * Additional 32 A100 80GB GPUs (4 nodes) in reserve\n\n    *  8 GPUs per node Using NVLink 4 inter-gpu connects, 4 OmniPath links\n\n    *   CPU: AMD\n\n    *   CPU memory: 512GB per node\n\n    *   GPU memory: 640GB per node\n\n    *   Inter-node connec","github_created_at":"2022-05-05T18:19:05+00:00","created_at":"2026-07-11T23:08:31.266307+00:00","updated_at":"2026-07-11T23:08:47.130857+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[],"trust":{"provenance":{"is_fork":false,"github_id":489082066,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:08:32.511Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1461,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:08:32.844Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:08:32.272Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T23:08:32.272Z"}}}}