readme: ---
  language: id
  widget:
  - text: "Penelitian ini bertujuan untuk menentukan identitas invertebrata laut dari Perairan Papua dengan teknik DNA barcoding"
  ---
  
  # Indonesian GPT-2 finetuned on Indonesian academic journals
  This is the [Indonesian gpt2-small model](https://huggingface.co/flax-community/gpt2-small-indonesian) fine-tuned to abstracts of Indonesian academic journals. All training was done on a TPUv2-8 VM sponsored by [TPU Research Cloud](https://sites.research.google/trc/).
  
  The demo can be found [here](https://huggingface.co/spaces/flax-community/gpt2-indonesian).
  
  ## How to use
  You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, 
  we set a seed for reproducibility:
  ```python
  >>> from transformers import pipeline, set_seed
  >>> generator = pipeline('text-generation', model='Galuh/id-journal-gpt2')
  >>> set_seed(42)
  >>> generator("Penelitian ini menggunakan teknik DNA barcoding untuk", max_length=30, num_return_sequences=5)
  
  [{'generated_text': 'Penelitian ini menggunakan teknik DNA barcoding untuk mendeteksi perubahan genetik bakteri pada udang windu. Empat tahap telah dilakukan, meliputi preparasi media untuk larva,'},
   {'generated_text': 'Penelitian ini menggunakan teknik DNA barcoding untuk identifikasi gen pengasil flavonoid.  Data yang diperoleh dari hasil PCR diidentifikasi dengan teknik sekuensing'},
   {'generated_text': 'Penelitian ini menggunakan teknik DNA barcoding untuk mengekstraksi fragmen DNA dari sampel kulit buaya dan tulang anjing, di mana proses ini melibatkan karakterisasi enzim yang'},
   {'generated_text': 'Penelitian ini menggunakan teknik DNA barcoding untuk melakukan transformasi. Tahapan transformasi meliputi seleksi sel dengan urutan (2, 8, 16,..., 18) dan'},
   {'generated_text': 'Penelitian ini menggunakan teknik DNA barcoding untuk amplifikasi genom DNA dengan menggunakan primer TG8226 dan TG806. Metode pol'}]
  ```
  
  Here is how to use this model to get the features of a given text in PyTorch:
  ```python
  from transformers import GPT2Tokenizer, GPT2Model
  tokenizer = GPT2Tokenizer.from_pretrained('Galuh/id-journal-gpt2')
  model = GPT2Model.from_pretrained('Galuh/id-journal-gpt2')
  text = "Ubah dengan teks apa saja."
  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)
  ```
  
  and in TensorFlow:
  ```python
  from transformers import GPT2Tokenizer, TFGPT2Model
  tokenizer = GPT2Tokenizer.from_pretrained('Galuh/id-journal-gpt2')
  model = TFGPT2Model.from_pretrained('Galuh/id-journal-gpt2')
  text = "Ubah dengan teks apa saja."
  encoded_input = tokenizer(text, return_tensors='tf')
  output = model(encoded_input)
  ```
  
  ## Limitations and bias  
  This model is originally the [Indonesian gpt2-small model](https://huggingface.co/flax-community/gpt2-small-indonesian), thus this model is also subject to the same [limitations and bias as the original model](https://huggingface.co/flax-community/gpt2-small-indonesian#limitations-and-bias). More detailed bias and analysis on this specific model is coming soon.
  
  ## Training data
  The model was trained on a dataset of Indonesian journals. We only trained this model on the abstracts. We extract the abstract by writing a script to find any text that is located between the word "Abstrak" (abstract) and "Kata kunci" (keywords). The extraction script can be found [here](https://github.com/galuhsahid/id-journal-gpt2/). To separate each abstract, we also add an end of text token (`<|endoftext|>`) between each abstract.
  
  The information of the sub-dataset and the distribution of the training and evaluation dataset are as follows:
  
  | split | count | percentage |
  | ---------- | ---------- | -------------- |
  | train    | 146,248      | 90%         |
  | validation    | 16,250      | 10%         |
  
  ## Training procedure 
  The model was trained on a TPUv2-8 VM provided by [TPU Research Cloud](https://sites.research.google/trc/). The training duration was `2h 30m 57s`.
  
  ### Evaluation results 
  The model achieves the following results without any fine-tuning (zero-shot):
  
  | dataset | train loss | eval loss | eval perplexity |
  | ---------- | ---------- | -------------- | ---------- |
  | Indonesian journals dataset (abstract only)    | 2.913      | 2.855         | 17.37   |
  
  ### Tracking
  The training process was tracked in [TensorBoard](https://huggingface.co/Galuh/id-journal-gpt2/tensorboard).