readme: ---
  language: ro
  tags:
  - bert
  - fill-mask
  mask_token: "[MASK]"
  license: mit
  ---
  
  The BERT **base**-**cased** model trained on all Romanian **tweets from 2008 to 2022**, version ![v2.0](https://img.shields.io/badge/v2-16%20Jan%202023-ff6666)
  
  
  # Versions
  
  - **ro-bert-tweet-v2**
    
    1.5 million steps of pre-training.
  
  - **ro-bert-tweet-v1**
    
    1 million steps of pre-training.
  
  - **ro-bert-tweet-v0**
    
    Initial version of the model after 300k steps of pre-training, only for setting up the framework.
  
  
  # How to use
  
  ```python
  import torch
  from transformers import AutoTokenizer, AutoModel
  
  # Load the tokenizer and the model
  tokenizer = AutoTokenizer.from_pretrained("Iulian277/ro-bert-tweet")
  model = AutoModel.from_pretrained("Iulian277/ro-bert-tweet")
  
  # Sanitize the input
  !pip install emoji
  from normalize import normalize # Use the `normalize` function from the `normalize.py` script
  normalized_text = normalize("Salut, ce faci?")
  
  # Tokenize the sentence and run through the model
  input_ids = torch.tensor(tokenizer.encode(normalized_text, add_special_tokens=True)).unsqueeze(0) # Batch size 1
  outputs = model(input_ids)
  
  # Get encoding
  last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple
  ```
  
  **Always** use the `normalize.py` script included in the repository to sanitize you input text, before feeding the tokenizer.
  Otherwise you will decrease the performance due to the [UNK] tokens.
  
  