readme: ---
  language:
  - tr
  thumbnail:
  tags:
  - gpt2
  - turkish
  
  license: apache-2.0
  datasets:
  - wikipedia-turkish
  metrics:
  - perplexity
  - accuracy
  
  widget:
  - text: Bu yazıyı bir bilgisayar yazdı. Yazarken
    context: ''
  - text: İnternete kolay erişim sayesinde dünya daha da küçüldü. Bunun sonucunda
    context: ''
  ---
  
  # Turkish GPT2 Model Finetuned 
  # Türkçe GPT2 Modeli
  
  ## Model description
  
  This is a GPT2-Small English based model finetuned and additionaly trainied with Wikipedia Articles in Turkish as of 28-10-2020
  
  Live demo based on this work at : https://www.metayazar.com/
  
  Fine tuned writer on this model: https://huggingface.co/gorkemgoknar/gpt2-turkish-writer
  
  Work has been done on Pierre Guillou tutorial as on this page.
  (https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) 
  
  Code is converted to work with Fastai 2.X .
  
  Using Google Colab for training. 
  
  Additional tutorial and source will be in https://github.com/gorkemgoknar in later stage.
  
  Current accuracy 33 %  , Perplexity : 51.88
  
  Models are available:
  
  * [gpt2-small-tuned-tr] (https://huggingface.co/gorkemgoknar/gpt2-small-turkish)
  * [gpt2-small-turkish-writer] (https://huggingface.co/gorkemgoknar/gpt2-turkish-writer)
  
  ## Intended uses & limitations
  
  #### How to use
  
  #### Install
  
  ```python
  from transformers import AutoTokenizer, AutoModelWithLMHead
  import torch
  
  tokenizer = AutoTokenizer.from_pretrained("gorkemgoknar/gpt2-small-turkish")
  model = AutoModelWithLMHead.from_pretrained("gorkemgoknar/gpt2-small-turkish")
  
  # Get sequence length max of 1024
  tokenizer.model_max_length=1024 
  
  model.eval()  # disable dropout (or leave in train mode to finetune)
  
  ```
  
  #### Generate 1 word
  ```python
  # input sequence
  text = "Bu yazıyı bilgisayar yazdı."
  inputs = tokenizer(text, return_tensors="pt")
  
  # model output
  outputs = model(**inputs, labels=inputs["input_ids"])
  loss, logits = outputs[:2]
  predicted_index = torch.argmax(logits[0, -1, :]).item()
  predicted_text = tokenizer.decode([predicted_index])
  
  # results
  print('input text:', text)
  print('predicted text:', predicted_text)
  
  # input text: 
  # predicted text:  
  
  ```
  
  #### Generate Full Sequence
  ```python
  # input sequence
  text = "Bu yazıyı bilgisayar yazdı."
  inputs = tokenizer(text, return_tensors="pt")
  
  # model output using Top-k sampling text generation method
  sample_outputs = model.generate(inputs.input_ids,
                                  pad_token_id=50256,
                                  do_sample=True, 
                                  max_length=50, # put the token number you want
                                  top_k=40,
                                  num_return_sequences=1)
  
  # generated sequence
  for i, sample_output in enumerate(sample_outputs):
      print(">> Generated text {}\\\\
  \\\\
  {}".format(i+1, tokenizer.decode(sample_output.tolist())))
  
  # >> Generated text
  #    
  
  ```
  
  #### Limitations and bias
  
  The training data used for this model come from Turkish Wikipedia. We know it contains a lot of unfiltered content from the internet, which is far from neutral. 
  
  
  ## Training data
  
  Wikipedia Turkish article dump as of 28-10-2020
  
  ## Training procedure
  
  
  ## Eval results
  
  | epoch\\\\t|train_loss\\\\t|valid_loss\\\\t|accuracy\\\\t|perplexity\\\\t|time   |
  | ----- | --------      |---------      | ----------    | ---------     | ----- |
  |0\\\\t|4.777015\\\\t|4.621834\\\\t|0.292547\\\\t|101.680367\\\\t|2:42:05|
  |1\\\\t|4.509412\\\\t|4.403999\\\\t|0.305574\\\\t|81.777267\\\\t|1:09:38|
  |2\\\\t|4.169529\\\\t|4.120755\\\\t|0.324908\\\\t|61.605747\\\\t|1:07:45|
  |3\\\\t|4.293973\\\\t|4.177899\\\\t|0.317211\\\\t|65.228653\\\\t|1:07:02|
  |4\\\\t|4.049848\\\\t|3.949103\\\\t|0.338347\\\\t|51.888783\\\\t|1:05:53|
  
  #Epoch 0 on Tesla T4, others on V100
  
  ```
  
  