readme: ---
  license: apache-2.0
  pipeline_tag: text-generation
  tags:
  - multilingual
  - PyTorch
  - Transformers
  - gpt3
  - gpt2
  - Deepspeed
  - Megatron
  datasets:
  - mc4
  - Wikipedia
  
  widget:
  - text: "I know you're tired, but can we go for another walk this evening?\npeter szemraj:\n\n" 
    example_title: "walk"
  - text: "What do you call an alligator who's just had surgery to remove his left arm?\npeter szemraj:\n\n" 
    example_title: "alligator"
  - text: "If you could live anywhere, where would it be?\npeter szemraj:\n\n" 
    example_title: "dream living place"
  - text: "What really makes you angry?\npeter szemraj:\n\n" 
    example_title: "pet peeve"
  - text: "My friend says that she knows every language, but she doesn't speak any of them.. what's wrong with her?\npeter szemraj:\n\n" 
    example_title: "language"
  - text: "What would you change about yourself if you could?\npeter szemraj:\n\n"
    example_title: "change"
  - text: "My first is in Asia, my second is in Europe, my third is in North America, and my fourth is in South America. What am I?\npeter szemraj:\n\n"
    example_title: "continent"
  - text: "Can you take me for dinner somewhere nice this time?\npeter szemraj:\n\n"
    example_title: "dinner"
  - text: "Honey, I have clogged the toilet for the third time this month.. sorry..\npeter szemraj:\n\n"
    example_title: "overflow"
  - text: "A man pushes his car to a hotel and tells the owner he's bankrupt. Why?\npeter szemraj:\n\n"
    example_title: "brain teaser"
  
  inference:
    parameters:
      min_length: 2
      max_length: 64
      length_penalty: 0.4
      no_repeat_ngram_size: 3
      do_sample: True
      top_p: 0.95
      top_k: 30
      temperature: 0.65
      repetition_penalty: 3.5
      
  ---
  
  
  # mGPT: fine-tune on message data MWE
  
  This model is a fine-tuned version of [sberbank-ai/mGPT](https://huggingface.co/sberbank-ai/mGPT) on 80k messages. Trained for one epoch, will be updated in a (separate) model repo later.
  
  ## Model description
  
  - testing if fine-tuned personality data bleeds over to other languages without being trained in them explicitly 
  
  ### Usage in python
  
  Install the transformers library if you don't have it:
  
  ```
  pip install -U transformers
  ```
  
  load the model into a pipeline object:
  
  
  ```
  from transformers import pipeline
  import torch
  device = 'cuda' if torch.cuda.is_available() else 'cpu'
  my_chatbot = pipeline('text-generation', 
                        'pszemraj/mGPT-Peter-mwe',
                        device=0 if device == 'cuda' else -1,
                      )
  ```
  
  
  
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 5e-05
  - train_batch_size: 4
  - eval_batch_size: 4
  - seed: 42
  - distributed_type: multi-GPU
  - gradient_accumulation_steps: 8
  - total_train_batch_size: 32
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine_with_restarts
  - lr_scheduler_warmup_ratio: 0.05
  - num_epochs: 1
  
  ### Framework versions
  
  - Transformers 4.18.0
  - Pytorch 1.11.0+cu113
  - Datasets 2.1.0
  - Tokenizers 0.12.1
  