readme: ---
  language: 
  - fr
  model-index:
  - name: asi/gpt-fr-cased-base
    results:
    - task:
        type: text-generation
        name: Wikitext-fr
      dataset:
        type: wikitext_fr
        name: Wikitext-fr
      metrics:
        - type: perplexity
          value: 109.2
          name: Perplexity
    - task:
        type: text-classification
        name: FLUE
      dataset:
        type: flue
        name: CLS-Books
        split: CLS
      metrics:
        - type: accuracy
          value: 88.3
          name: Accuracy
    - task:
        type: text-classification
        name: FLUE
      dataset:
        type: flue
        name: CLS-Dvd
        split: CLS
      metrics:
        - type: accuracy
          value: 86.9
          name: Accuracy
    - task:
        type: text-classification
        name: FLUE
      dataset:
        type: flue
        name: CLS-Music
        split: CLS
      metrics:
        - type: accuracy
          value: 89.3
          name: Accuracy
    - task:
        type: text-classification
        name: FLUE
      dataset:
        type: flue
        name: PAWS-X
        split: PAWS-X
      metrics:
        - type: accuracy
          value: 83.3
          name: Accuracy
    - task:
        type: text-classification
        name: FLUE
      dataset:
        type: flue
        name: XNLI
        split: XNLI
      metrics:
        - type: accuracy
          value: 75.6
          name: Accuracy
    - task:
        type: summarization
        name: OrangeSum
      dataset:
        type: orange_sum
        name: OrangeSum-Abstract
        split: abstract
      metrics:
      - name: ROUGE-1
        type: rouge
        value: 17.5
      - name: ROUGE-2
        type: rouge
        value: 3.1
      - name: ROUGE-L
        type: rouge
        value: 12.1
    - task:
        type: summarization
        name: OrangeSum
      dataset:
        type: orange_sum
        name: OrangeSum-Title
        split: title
      metrics:
      - name: ROUGE-1
        type: rouge
        value: 13.9
      - name: ROUGE-2
        type: rouge
        value: 2.3
      - name: ROUGE-L
        type: rouge
        value: 9.7
  tags:
  - tf
  - pytorch
  - gpt2
  - text-generation
  license: apache-2.0
  thumbnail: https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png
  ---
  
  <img src="https://raw.githubusercontent.com/AntoineSimoulin/gpt-fr/main/imgs/logo.png" width="200">
  
  ## Model description
  
  **GPT-fr** 🇫🇷 is a GPT model for French developped by [Quantmetry](https://www.quantmetry.com/) and the [Laboratoire de Linguistique Formelle (LLF)](http://www.llf.cnrs.fr/en). We train the model on a very large and heterogeneous French corpus. We release the weights for the following configurations:
  
  | Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
  | :------:       |   :---: | :---: | :---: | :---: |
  | `gpt-fr-cased-small` | 12    | 12    | 768   | 124 M |
  | `gpt-fr-cased-base` | 24    | 14    | 1,792   | 1,017 B |
  
  ## Intended uses & limitations
  
  The model can be leveraged for language generation tasks. Besides, many tasks may be formatted such that the output is directly generated in natural language. Such configuration may be used for tasks such as automatic summary or question answering. We do hope our model might be used for both academic and industrial applications. 
  
  #### How to use
  
  The model might be used through the astonishing 🤗 `Transformers` librairie:
  
  ```python
  from transformers import GPT2Tokenizer, GPT2LMHeadModel
  
  # Load pretrained model and tokenizer
  model = GPT2LMHeadModel.from_pretrained("asi/gpt-fr-cased-small")
  tokenizer = GPT2Tokenizer.from_pretrained("asi/gpt-fr-cased-small")
  
  # Generate a sample of text
  model.eval()
  input_sentence = "Longtemps je me suis couché de bonne heure."
  input_ids = tokenizer.encode(input_sentence, return_tensors='pt')
  
  beam_outputs = model.generate(
      input_ids, 
      max_length=100, 
      do_sample=True,   
      top_k=50, 
      top_p=0.95, 
      num_return_sequences=1
  )
  
  print("Output:\n" + 100 * '-')
  print(tokenizer.decode(beam_outputs[0], skip_special_tokens=True))
  ```
  
  #### Limitations and bias
  
  Large language models tend to replicate the biases found in pre-training datasets, such as gender discrimination or offensive content generation.
  
  To limit exposition to too much explicit material, we carefully choose the sources beforehand. This process — detailed in our paper — aims to limit offensive content generation from the model without performing manual and arbitrary filtering.
  
  However, some societal biases, contained in the data, might be reflected by the model. For example on gender equality, we generated the following sentence sequence "Ma femme/Mon mari vient d'obtenir un nouveau poste. A partir de demain elle/il sera \_\_\_\_\_\_\_" and observed the model generated distinct positions given the subject gender. We used top-k random sampling strategy with k=50 and stopped at the first punctuation element.
  The positions generated for the wife is '_femme de ménage de la maison_' while the position for the husband is '_à la tête de la police_'. We do appreciate your feedback to better qualitatively and quantitatively 