pytorch jax gpt2 lm-head causal-lm id dataset:cc100 transformers text-generation
.gitattributes README.md config.json flax_model.msgpack merges.txt pytorch_model.bin vocab.json
---
language: "id"
datasets:
- cc100
widget:
- text: "Pada suatu hari, Ilmuwan menemukan hewan unik. Hewan ini"
---
# Indonesian Small GPT model

IndoGPT-small is a transformers model pretrained on Indonesian CC100 in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to guess the next word in sentences.

This model was trained for xx steps on Indonesian CC100. Using [GPT-Neo](https://github.com/EleutherAI/gpt-neo) from EleutherAI

CC100 is not the best dataset to train GPT. There are a lot of boilerplate content, etc. But, It is one of the largest dataset for Indonesian. So, I use this dataset.

## How to use

You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:

```py
>>> from transformers import pipeline
>>> generator = pipeline('text-generation', model='Wikidepia/IndoGPT-small')
>>> generator("Kota bali adalah", do_sample=True, min_length=50)

[{'generated_text': 'Kota bali adalah satu diantara pulau yang memiliki keindahan dan keunikan budaya dan warisan budaya yang tinggi.'}]
```

## Acknowledgement

Thanks to Tensorflow Research Cloud for providing TPU v3-8s.
