readme:  ---
  inference: false
  language: id
  ---
  
  # IndoConvBERT Base Model
  
  IndoConvBERT is a ConvBERT model pretrained on Indo4B.
  
  ## Pretraining details
  
  We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 512 sequence length, we pre-train the model with 512 sequence length for 1M steps on a v3-8 TPU.
  
  The current version of the model is trained on Indo4B and small Twitter dump.
  
  ## Acknowledgement
  
  Big thanks to TFRC (TensorFlow Research Cloud) for providing free TPU.
  