readme: ---
  license: mit
  datasets:
  - sagawa/ZINC-canonicalized
  metrics:
  - accuracy
  model-index:
  - name: ZINC-deberta
    results:
    - task:
        name: Masked Language Modeling
        type: fill-mask
      dataset:
        name: sagawa/ZINC-canonicalized
        type: sagawa/ZINC-canonicalized
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.9497212171554565
  ---
  
  # ZINC-t5 
  
  This model is a fine-tuned version of [google/t5-v1_1-base](https://huggingface.co/microsoft/deberta-base) on the sagawa/ZINC-canonicalized dataset.
  It achieves the following results on the evaluation set:
  - Loss: 0.1202
  - Accuracy: 0.9497
  
  
  ## Model description
  
  We trained t5 on SMILES from ZINC using the task of masked-language modeling (MLM). Its tokenizer is also trained on ZINC.
  
  
  ## Intended uses & limitations
  
  This model can be used for the prediction of molecules' properties, reactions, or interactions with proteins by changing the way of finetuning.
  As an example, We finetuned this model to predict products. The model is [here](https://huggingface.co/sagawa/ZINC-t5-productpredicition), and you can use the demo [here](https://huggingface.co/spaces/sagawa/predictproduct-t5).
  Using its encoder, we trained a regression model to predict a reaction yield. You can use this demo [here](https://huggingface.co/spaces/sagawa/predictyield-t5).
  
  ## Training and evaluation data
  
  We downloaded [ZINC data](https://drive.google.com/drive/folders/1lSPCqh31zxTVEhuiPde7W3rZG8kPgp-z) and canonicalized them using RDKit. Then, we dropped duplicates. The total number of data is 22992522, and they were randomly split into train:validation=10:1.
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 5e-03
  - train_batch_size: 30
  - eval_batch_size: 32
  - seed: 42
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - num_epochs: 30.0
  
  ### Training results
  
  | Training Loss | Step   | Accuracy | Validation Loss |
  |:-------------:|:------:|:--------:|:---------------:|
  | 0.2471        | 25000  | 0.9843   | 0.2226          |
  | 0.1871        | 50000  | 0.9314   | 0.1783          |
  | 0.1791        | 75000  | 0.9371   | 0.1619          |
  | 0.1596        | 100000 | 0.9401   | 0.1520          |
  | 0.1522        | 125000 | 0.9422   | 0.1449          |
  | 0.1435        | 150000 | 0.9436   | 0.1404          |
  | 0.1421        | 175000 | 0.9447   | 0.1368          |
  | 0.1398        | 200000 | 0.9459   | 0.1322          |
  | 0.1297        | 225000 | 0.9466   | 0.1299          |
  | 0.1324        | 250000 | 0.9473   | 0.1268          |
  | 0.1257        | 275000 | 0.9483   | 0.1244          |
  | 0.1266        | 300000 | 0.9491   | 0.1216          |
  | 0.1301        | 325000 | 0.9497   | 0.1204          |