readme: ---
  license: bigscience-bloom-rail-1.0
  language:
  - ak
  - ar
  - as
  - bm
  - bn
  - ca
  - code
  - en
  - es
  - eu
  - fon
  - fr
  - gu
  - hi
  - id
  - ig
  - ki
  - kn
  - lg
  - ln
  - ml
  - mr
  - ne
  - nso
  - ny
  - or
  - pa
  - pt
  - rn
  - rw
  - sn
  - st
  - sw
  - ta
  - te
  - tn
  - ts
  - tum
  - tw
  - ur
  - vi
  - wo
  - xh
  - yo
  - zh
  - zhs
  - zht
  - zu
  ---
  
  # <span style="color:red"><b>WARNING:</b> The checkpoints on this repo are not fully trained model. Evaluations of intermediary checkpoints and the final model will be added when conducted (see below).</span>
  
  # <p>BLOOM LM<br/> _BigScience Large Open-science Open-access Multilingual Language Model_ <br/>Model Card</p>
  <img src="https://assets.website-files.com/6139f3cdcbbff3a68486761d/613cd8997b270da063e230c5_Tekengebied%201-p-500.png" alt="BigScience Logo" width="200"/>
  
  
  Version 1.3 / 11.July.2022 - Available intermediary checkpoints - global steps: 
  
  + `1000`, `10000`, `100000`, `200000`, `300000`, `400000`, `500000`, `600000`
  
  You can check the available checkpoints by clicking on the branches section of the repo
  
  # How to load a specific version
  
  We use `git tags` to load a model in a specific version (eg. `global_step1000`):
  
  ```python
  from transformers import AutoModelForCausalLM
  model = AutoModelForCausalLM.from_pretrained(
  	"bigscience/bloom-350m-intermediate",
  	revision="global_step1000",
  	torch_dtype="auto",
  )
  ```
  
  
  
  # Table of Contents
  1. [Model Details](#model-details)
  2. [Uses](#uses)
  3. [Training Data](#training-data)
  4. [Risks and Limitations](#risks-and-limitations)
  5. [Evaluation](#evaluation)
  6. [Recommendations](#recommendations)
  7. [Glossary and Calculations](#glossary-and-calculations)
  8. [More Information](#more-information)
  9. [Model Card Authors](#model-card-authors)
  
  ---
  
  # Model Details  
  
  BLOOM is a type of language model, which is a probability distribution over sequences of words.  Specifically, BLOOM is a Large Language Model (LLM), meaning that it is trained on vast amounts of text data using industrial-scale computational resources.  As such, the model is able to capture the statistical tendencies of words, phrases, sentences, and larger spans of text that it is exposed to in the training data.
  
  ## Basics
  *This section provides information about the model type, version, license, funders, release date, developers, and contact information.*
  *It is useful for anyone who wants to reference the model.*
  
  <details>
  <summary>Click to expand</summary>
    
  **Developed by:** BigScience ([website](https://bigscience.huggingface.co))
  
  *All collaborators are either volunteers or have an agreement with their employer. (Further breakdown of participants forthcoming.)*
      
  **Model Type:** Transformer-based Language Model
  
  **Version:** 1.0.0
  
  **Languages:** Multiple; see [training data](#training-data)
  
  **License:** RAIL License v1.0 ([link](https://huggingface.co/spaces/bigscience/license))
  
  **Release Date Estimate:** Monday, 11.July.2022
  
  **Send Questions to:** bigscience-contact@googlegroups.com
  
  **Cite as:** BigScience, _BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model_. International, May 2021-May 2022
  
  **Funded by:** 
      
  * The French government.
  
  * Hugging Face ([website](https://huggingface.co)).
  
  * Organizations of contributors.  *(Further breakdown of organizations forthcoming.)*
  
  </details>
  
  ## Technical Specifications
  *This section includes details about the model objective and architecture, and the compute infrastructure.*
  *It is useful for people interested in model development.*
  
  <details>
  <summary>Click to expand</summary>
  
  Please see [the BLOOM training README](https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml#readme) for full details on replicating training.
  
  ### Model Architecture and Objective
  
  * Modified from Megatron-LM GPT2 (see [paper](https://arxiv.org/abs/1909.08053), [BLOOM Megatron code](https://github.com/bigscience-workshop/Megatron-DeepSpeed)):
  
  * Decoder-only architecture
  
  * Layer normalization applied to word embeddings layer (`StableEmbedding`; see [code](https://github.com/facebookresearch/bitsandbytes), [paper](https://arxiv.org/pdf/2110.02861.pdf))
  
  * ALiBI positional encodings (see [paper](https://arxiv.org/pdf/2108.12409.pdf)), with GeLU activation functions
  
  * 176 billion parameters:
  
      * 70 layers, 112 attention heads
  
      * Hidden layers are 14336-dimensional
  
      * Sequence length of 2048 tokens used (see [BLOOM tokenizer](https://huggingface.co/bigscience/tokenizer), [tokenizer description](#tokenization))
  
  **Objective Function:** Cross Entropy with mean reduction (see [API documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss)).
      
  ### Compute infrastructure
  Jean Zay Public Supercomputer, provided by the French government (see [announcement](https://www.enseignementsup-recherche.gouv.fr/fr/signature-du-marche-d-acquisition-de-l-un-des-supercalculateurs-les-plus-puissants-d-europe-46733)).
  
  #### Hardware
  
  * 384 A100 80GB GPUs (