readme: ---
  language:
  - fr
  license: apache-2.0
  ---
  
  Remark: I'm an independant student. This model was made for personnal use. It may not be perfect.
  
  ## Model description
  
  **GPTheo-fr** 🇫🇷 is a GPT model in French for theology based on the asi/gpt-fr-cased-small model.
  
  | Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
  | :------:       |   :---: | :---: | :---: | :---: |
  | `gptheo-fr` | 12    | 12    | 768   | 124 M |
  
  ## Intended uses & limitations
  
  The model can be leveraged for language generation tasks. It is fine-tuned for theology.
  
  #### How to use
  
  The model might be used through the astonishing 🤗 `Transformers` librairie:
  
  ```python
  from transformers import GPT2Tokenizer
  
  # Load pretrained model and tokenizer
  model = torch.load('pytorch_model.bin')
  tokenizer = GPT2Tokenizer.from_pretrained("asi/gpt-fr-cased-small")
  
  # Generate a sample of text
  model.eval()
  input_sentence = "Dieu est-il une composante importante de nos vies?"
  input_ids = tokenizer.encode(input_sentence, return_tensors='pt')
  
  beam_outputs = model.generate(
      input_ids, 
      max_length=100, 
      do_sample=True,
      num_beams=5,   
      top_k=50, 
      top_p=0.95, 
      num_return_sequences=1,
      repetition_penalty=10.0
  )
  
  print("Output:\n" + 100 * '-')
  print(tokenizer.decode(beam_outputs[0], skip_special_tokens=True))
  ```
  
  ## Training data
  
  I created a custom dataset from french theological theses. It trained for 10k steps on my custom data on top of all the training done by the original asi/gpt-fr-cased-small.
  