readme: ---
  license: bigscience-bloom-rail-1.0
  language:
  - ak
  - ar
  - as
  - bm
  - bn
  - ca
  - code
  - en
  - es
  - eu
  - fon
  - fr
  - gu
  - hi
  - id
  - ig
  - ki
  - kn
  - lg
  - ln
  - ml
  - mr
  - ne
  - nso
  - ny
  - or
  - pa
  - pt
  - rn
  - rw
  - sn
  - st
  - sw
  - ta
  - te
  - tn
  - ts
  - tum
  - tw
  - ur
  - vi
  - wo
  - xh
  - yo
  - zh
  - zhs
  - zht
  - zu
  pipeline_tag: text-generation
  ---
  
  <h1 style='text-align: center '>BLOOM LM</h1> 
  <h2 style='text-align: center '><em>BigScience Large Open-science Open-access Multilingual Language Model</em> </h2> 
  <h3 style='text-align: center '>Model Card</h3>
  <img src="https://s3.amazonaws.com/moonup/production/uploads/1657124309515-5f17f0a0925b9863e28ad517.png" alt="BigScience Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
  
  Version 1.0 / 26.May.2022
  
  ## Table of Contents
  1. [Model Details](#model-details)
  2. [Uses](#uses)
  3. [Training Data](#training-data)
  4. [Risks and Limitations](#risks-and-limitations)
  5. [Evaluation](#evaluation)
  6. [Recommendations](#recommendations)
  7. [Glossary and Calculations](#glossary-and-calculations)
  8. [More Information](#more-information)
  9. [Model Card Authors](#model-card-authors)
  
  ## Model Details  
  
  ### Basics
  *This section provides information for anyone who wants to know about the model.*
  
  <details>
  <summary>Click to expand</summary> <br/>
      
  **Developed by:** BigScience ([website](https://bigscience.huggingface.co))
  
  * All collaborators are either volunteers or have an agreement with their employer. *(Further breakdown of participants forthcoming.)*
      
  **Model Type:** Transformer-based Language Model
  
  **Version:** 1.0.0
  
  **Languages:** Multiple; see [training data](#training-data)
  
  **License:** RAIL License v1.0 ([link](https://huggingface.co/spaces/bigscience/license))
  
  **Release Date Estimate:** Monday, 11.July.2022
  
  **Send Questions to:** bigscience-contact@googlegroups.com
  
  **Cite as:** BigScience, _BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model_. International, May 2021-May 2022
  
  **Funded by:** 
      
  * The French government.
  
  * Hugging Face ([website](https://huggingface.co)).
  
  * Organizations of contributors.  *(Further breakdown of organizations forthcoming.)*
  
  </details>
  
  ### Technical Specifications
  *This section provides information for people who work on model development.*
  
  <details>
  <summary>Click to expand</summary><br/>
  
  Please see [the BLOOM training README](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml#readme) for full details on replicating training.
  
  **Model Architecture:** Modified from Megatron-LM GPT2 (see [paper](https://arxiv.org/abs/1909.08053), [BLOOM Megatron code](https://github.com/bigscience-workshop/Megatron-DeepSpeed)):
  
  * Decoder-only architecture
  
  * Layer normalization applied to word embeddings layer (`StableEmbedding`; see [code](https://github.com/facebookresearch/bitsandbytes), [paper](https://arxiv.org/pdf/2110.02861.pdf))
  
  * ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions
  
  * 350 million parameters:
  
      * 24 layers, 16 attention heads
  
      * Hidden layers are 1024-dimensional
  
      * Sequence length of 2048 tokens used (see [BLOOM tokenizer](https://huggingface.co/bigscience/tokenizer), [tokenizer description](#tokenization))
  
  **Objective Function:** Cross Entropy with mean reduction (see [API documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss)).
      
  **Compute infrastructure:** Jean Zay Public Supercomputer, provided by the French government (see [announcement](https://www.enseignementsup-recherche.gouv.fr/fr/signature-du-marche-d-acquisition-de-l-un-des-supercalculateurs-les-plus-puissants-d-europe-46733)).
  
  * Hardware: 384 A100 80GB GPUs (48 nodes):
      
      * Additional 32 A100 80GB GPUs (4 nodes) in reserve
  
      *  8 GPUs per node Using NVLink 4 inter-gpu connects, 4 OmniPath links
  
      *   CPU: AMD
  
      *   CPU memory: 512GB per node
  
      *   GPU memory: 640GB per node
  
      *   Inter-node connect: Omni-Path Architecture (OPA)
  
      *   NCCL-communications network: a fully dedicated subnet
  
      *   Disc IO network: shared network with other types of nodes
  
  * Software:
    
      *   Megatron-DeepSpeed ([Github link](https://github.com/bigscience-workshop/Megatron-DeepSpeed))
  
      *   DeepSpeed ([Github link](https://github.com/microsoft/DeepSpeed))
  
      *   PyTorch (pytorch-1.11 w/ CUDA-11.5; see [Github link](https://github.com/pytorch/pytorch))
  
      *   apex ([Github link](https://github.com/NVIDIA/apex))
  
  
  #### **Training**
  
   
  _In progress._
  
  Current training logs: [Tensorboard link](https://huggingface.co/tensorboard/bigscience/tr11-176B-ml-logs/)
  
  - Checkpoint size:
      
      - Bf16 weights: 329GB
      
      - Full checkpoint with optimizer states: 2.3TB
  
  - Training throughput: About 150 TFLOP per GPU per second
  
  - Number of epochs: 1 (*current target*)
  
  - Dates:
      
      - Started 11th March, 2022 11:42am PST
  
      - 