readme: # Material SciBERT (TPU): Improving language understanding in materials science
  
  **Work in progress**
  
  ## Introduction 
  SciBERT-based model pre-trained with materials science scientific fulltext
  
  ## Authors
  Luca Foppiano
  Pedro Ortiz Suarez
  
  ## TLDR
  - Collected full-text from ~700000 articles provided by the National Institute for Materials Science (NIMS) TDM platform (https://dice.nims.go.jp/services/TDM-PF/en/), dataset called ScienceCorpus (SciCorpus)
  - We added to the SciBERT vocabulary (32k tokens), 100 domain-specific unknown words extracted from SciCorpus with a keywords modeler (KeyBERT)
  - Starting conditions: original SciBERT weights 
  - Pre-train the model MatTpuSciBERT from on the Google Cloud with the TPU (Tensor Processing Unit) as follow:
    - 800000 steps with batch_size: 256, max_seq_length:512 
    - 100000 steps with batch_size: 2048, max_seq_length:128
  - Fine-tuning and testing on NER on superconductors (https://github.com/lfoppiano/grobid-superconductors) and physical quantities (https://github.com/kermitt2/grobid-quantities)
  
  ## Related work
  
  ### BERT Implementations
  - BERT (the original) https://arxiv.org/abs/1810.04805
  - RoBERTa (Re-implementation by Facebook) https://arxiv.org/abs/1907.11692
  
  ### Relevant models
  - SciBERT: BERT, from scratch, scientific articles (biology + CS) https://github.com/allenai/scibert
  - MatSciBERT (Gupta): RoBERTa, from scratch, SciBERT vocab and weights, ~150 K paper limited to 4 MS families http://github.com/m3rg-iitd/matscibert
  - MaterialBERT: Not yet published
  - MatBERT (CEDER): BERT, from scratch, 2M documents on materials science  (~60M paragraphs) https://github.com/lbnlp/MatBERT
  - BatteryBERT (Cole): BERT, mixed from scratch and with predefined weights https://github.com/ShuHuang/batterybert/
  
  ### Results 
  
  Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/kermitt2/delft)
  
  #### NER Superconductors
  
  | Model                | Precision | Recall  | F1     |
  |----------------------|-----------|---------|--------|
  | SciBERT (baseline)   | 81.62%    | 84.23%  | 82.90% |
  | MatSciBERT (Gupta)   | 81.45%    | 84.36%  | 82.88% |
  | MatTPUSciBERT        | 82.13%    | 85.15%  | 83.61% |
  | MatBERT (Ceder)      | 81.25%    | 83.99%  | 82.60% |
  | BatteryScibert-cased | 81.09%    | 84.14%  | 82.59% |
  
  #### NER Quantities
  
  | Model                | Precision | Recall  | F1       |
  |----------------------|-----------|---------|----------|
  | SciBERT (baseline)   | 88.73%    | 86.76%  | 87.73%   |
  | MatSciBERT (Gupta)   | 84.98%    | 90.12%  | 87.47%   |
  | MatTPUSciBERT        | 88.62%    | 86.33%  | 87.46%   |
  | MatBERT (Ceder)      | 85.08%    | 89.93%  | 87.44%   |
  | BatteryScibert-cased | 85.02%    | 89.30%  | 87.11%   |
  | BatteryScibert-cased | 81.09%    | 84.14%  | 82.59%   |
     
  
  ## References
  
  This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers 
  
  ## Acknowledgements
  
  TBA