readme: ---
  language:
  - ru
  - en
  pipeline_tag: text2text-generation
  tags:
  - PyTorch
  - Transformers
  - gpt2
  - squad
  - lm-head
  - casual-lm
  thumbnail: "https://github.com/RussianNLP/RusEnQA"
  
  ---
  
  ## RusEnQA
  
  QA for Russian and English based on the [rugpt3xl](https://huggingface.co/sberbank-ai/rugpt3xl) model
  
  ### Fine-tuning format:
  
  ```
   "<s>paragraph: "+eng_context+"\nlang: rus\nquestion: "+rus_question+' answer: '+ rus_answer+"</s>"
  ```
  
  ### About ruGPT-3 XL model
  Model was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [SberDevices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  
  *Note! Model has sparse attention blocks.*
  
  Total training time was around 10 days on 256 GPUs.
  Final perplexity on test set is 12.05. Model parameters: 1.3B.