readme: ---
  language: "en"
  tags:
  - twitter
  - masked-token-prediction
  - election2020
  - politics
  license: "gpl-3.0"
  ---
  
  # Pre-trained BERT on Twitter US Political Election 2020
  
  Pre-trained weights for [Knowledge Enhance Masked Language Model for Stance Detection](https://www.aclweb.org/anthology/2021.naacl-main.376), NAACL 2021.
  
  We use the initialized weights from BERT-base (uncased) or `bert-base-uncased`.
  
  # Training Data
  
  This model is pre-trained on over 5 million English tweets about the 2020 US Presidential Election.
  
  # Training Objective
  
  This model is initialized with BERT-base and trained with normal MLM objective.
  
  # Usage
  
  This pre-trained language model **can be fine-tunned to any downstream task (e.g. classification)**.
  
  Please see the [official repository](https://github.com/GU-DataLab/stance-detection-KE-MLM) for more detail.
  
  ```python
  from transformers import BertTokenizer, BertForMaskedLM, pipeline
  import torch
  
  # Choose GPU if available
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  
  # Select mode path here
  pretrained_LM_path = "kornosk/bert-political-election2020-twitter-mlm"
  
  # Load model
  tokenizer = BertTokenizer.from_pretrained(pretrained_LM_path)
  model = BertForMaskedLM.from_pretrained(pretrained_LM_path)
  
  # Fill mask
  example = "Trump is the [MASK] of USA"
  fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
  # Use following line instead of the above one does not work.
  # Huggingface have been updated, newer version accepts a string of model name instead.
  fill_mask = pipeline('fill-mask', model=pretrained_LM_path, tokenizer=tokenizer)
  
  outputs = fill_mask(example)
  print(outputs)
  
  # See embeddings
  inputs = tokenizer(example, return_tensors="pt")
  outputs = model(**inputs)
  print(outputs)
  
  # OR you can use this model to train on your downstream task!
  # Please consider citing our paper if you feel this is useful :)
  ```
  
  # Reference
  
  - [Knowledge Enhance Masked Language Model for Stance Detection](https://www.aclweb.org/anthology/2021.naacl-main.376), NAACL 2021.
  
  # Citation
  ```bibtex
  @inproceedings{kawintiranon2021knowledge,
      title={Knowledge Enhanced Masked Language Model for Stance Detection},
      author={Kawintiranon, Kornraphop and Singh, Lisa},
      booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
      year={2021},
      publisher={Association for Computational Linguistics},
      url={https://www.aclweb.org/anthology/2021.naacl-main.376}
  }
  ```