readme: # Model description
  This model is BERT-based architecture with 8 layers. The detailed config is summarized as follows. The drug-like molecule BERT is inspired by ["Self-Attention Based Molecule Representation for Predicting Drug-Target Interaction"](https://arxiv.org/abs/1908.06760). We modified several points of training procedures.
  
  ```
  config = BertConfig(
      vocab_size=vocab_size,
      hidden_size=128,
      num_hidden_layers=8,
      num_attention_heads=8,
      intermediate_size=512,
      hidden_act="gelu",
      hidden_dropout_prob=0.1,
      attention_probs_dropout_prob=0.1,
      max_position_embeddings=max_seq_len + 2,
      type_vocab_size=1,
      pad_token_id=0,
      position_embedding_type="absolute"
  )
  ```
  
  # Training and evaluation data
  It's trained on drug-like molecules on the PubChem database. The PubChem database contains more than 100 M molecules, therefore, we filtered drug-like molecules using the quality of drug-likeliness score (QED). The 4.1 M molecules were filtered and the QED score threshold was set to 0.7. 
  
  # Tokenizer
  We utilize a character-level tokenizer. The special tokens are "[SOS]", "[EOS]", "[PAD]", "[UNK]".
  
  # Training hyperparameters
  The following hyperparameters were used during training:
  - Adam optimizer, learning_rate: 5e-4, scheduler: cosine annealing 
  - Batch size: 2048
  - Training steps: 24 K
  - Training_precision: FP16
  - Loss function: cross-entropy loss
  - Training masking rate: 30 %
  - Testing masking rate: 15 % (original molecule BERT utilized 15 % of masking rate)
  - NSP task: None
  
  # Performance
  - Accuracy: 94.02 %