readme: ---
  language: ar
  ---
  
  # Sanaa
  ## Arabic GPT-2 demo
  
  This is a small GPT-2 model retrained on Arabic Wikipedia circa September 2020
  (due to memory limits, the first 600,000 lines of the Wiki dump)
  
  There is NO content filtering in the current version; do not use for public-facing
  text generation.
  
  ## Training
  
  Training notebook: https://colab.research.google.com/drive/1Z_935vTuZvbseOsExCjSprrqn1MsQT57
  
  Steps to training:
  - Follow beginning of Pierre Guillou's Portuguese GPT-2 notebook: https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb to download Arabic Wikipedia and run WikiExtractor
  - Read Beginner's Guide by Ng Wai Foong https://medium.com/@ngwaifoong92/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f
  - Following Ng Wai Foong's instructions, create an encoded .npz corpus (this was very small in my project
     and would be improved by adding many X more training data)
  - Run generate_unconditional_samples.py and other sample code to generate text
  - Download TensorFlow checkpoints
  - Use my notebook code to write vocab.json, empty merge.txt
  - Copy config.json from similar GPT-2 arch, edit for changes as needed
  
  ```python
  am = AutoModel.from_pretrained('./argpt', from_tf=True)
  am.save_pretrained("./")
  ```
  
  ## Generating text in SimpleTransformers
  
  Finetuning notebook: https://colab.research.google.com/drive/1fXFH7g4nfbxBo42icI4ZMy-0TAGAxc2i
  
  ```python
  from simpletransformers.language_generation import LanguageGenerationModel
  model = LanguageGenerationModel("gpt2", "monsoon-nlp/sanaa")
  model.generate("مدرستي")
  ```
  
  ## Finetuning dialects in SimpleTransformers
  
  I finetuned this model on different Arabic dialects to generate a new
  model (monsoon-nlp/sanaa-dialect on HuggingFace) with some additional
  control tokens.
  
  Finetuning notebook: https://colab.research.google.com/drive/1fXFH7g4nfbxBo42ic$
  
  ```python
  from simpletransformers.language_modeling import LanguageModelingModel
  ft_model = LanguageModelingModel('gpt2', 'monsoon-nlp/sanaa', args=train_args)
  ft_model.tokenizer.add_tokens(["[EGYPTIAN]", "[MSA]", "[LEVANTINE]", "[GULF]"])
  ft_model.model.resize_token_embeddings(len(ft_model.tokenizer))
  ft_model.train_model("./train.txt", eval_file="./test.txt")
  
  # exported model
  from simpletransformers.language_generation import LanguageGenerationModel
  model = LanguageGenerationModel("gpt2", "./dialects")
  model.generate('[EGYPTIAN]' + "مدرستي")
  ```
  