GraphCanon
Home/Model Training/model_card
model_card logo

model_card

Enrichment pending
bigscience-workshop/model_card

model_card

GitHub

GraphCanon updated today · GitHub synced today

26
Stars
5
Forks
0
Open issues
2
Watchers
4y
Last push
Apache-2.0Created May 5, 2022

Trust & integrity

Full report
Maintenance
Dormant (1461d since push)
As of today · Source: github_public_v1
Provenance
Not a fork · Organization account
As of today · Source: github_public_v1
Security (OSV)
No lockfile
As of today · Source: none

Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.

Capability facts

No sourced capability facts yet. Facts appear after ingest scans repo manifests (Dockerfile, package.json, MCP configs).

Categories

Model TrainingLLM FrameworksVector Databases

README

license: bigscience-bloom-rail-1.0 language:

  • ak
  • ar
  • as
  • bm
  • bn
  • ca
  • code
  • en
  • es
  • eu
  • fon
  • fr
  • gu
  • hi
  • id
  • ig
  • ki
  • kn
  • lg
  • ln
  • ml
  • mr
  • ne
  • nso
  • ny
  • or
  • pa
  • pt
  • rn
  • rw
  • sn
  • st
  • sw
  • ta
  • te
  • tn
  • ts
  • tum
  • tw
  • ur
  • vi
  • wo
  • xh
  • yo
  • zh
  • zhs
  • zht
  • zu

DEPRECATED. This is now live at https://huggingface.co/bigscience/bloom . Please make additional changes there!

BLOOM LM
BigScience Large Open-science Open-access Multilingual Language Model
Model Card

BigScience Logo

Version 1.0 / 25.May.2022

Table of Contents

  1. Model Details
  2. Uses
  3. Training Data
  4. Risks and Limitations
  5. Evaluation
  6. Recommendations
  7. Glossary and Calculations
  8. More Information
  9. Model Card Authors

Model Details

Basics

This section provides information for anyone who wants to know about the model.

Click to expand

Developed by: BigScience (website)

  • All collaborators are either volunteers or have an agreement with their employer. (Further breakdown of participants forthcoming.)

Model Type: Transformer-based Language Model

Version: 1.0.0

Languages: Multiple; see training data

License: RAIL License v1.0 (link)

Release Date Estimate: Monday, 11.July.2022

Send Questions to: bigscience-contact@googlegroups.com

Cite as: BigScience, BigScience Language Open-source Open-access Multilingual (BLOOM) Language Model. International, May 2021-May 2022

Funded by:

  • The French government.

  • Hugging Face (website).

  • Organizations of contributors. (Further breakdown of organizations forthcoming.)

Technical Specifications

This section provides information for people who work on model development.

Click to expand

Please see the BLOOM training README for full details on replicating training.

Model Architecture: Modified from Megatron-LM GPT2 (see paper, BLOOM Megatron code):

  • Decoder-only architecture

  • Layer normalization applied to word embeddings layer (StableEmbedding; see code, paper)

  • ALiBI positional encodings (see paper), with GeLU activation functions

  • 176 billion parameters:

    • 70 layers, 112 attention heads

    • Hidden layers are 14336-dimensional

    • Sequence length of 2048 tokens used (see BLOOM tokenizer, tokenizer description)

Objective Function: Cross Entropy with mean reduction (see API documentation).

Compute infrastructure: Jean Zay Public Supercomputer, provided by the French government (see announcement).

  • Hardware: 384 A100 80GB GPUs (48 nodes):

    • Additional 32 A100 80GB GPUs (4 nodes) in reserve

    • 8 GPUs per node Using NVLink 4 inter-gpu connects, 4 OmniPath links

    • CPU: AMD

    • CPU memory: 512GB per node

    • GPU memory: 640GB per node

    • Inter-node connec