readme: ---
  tags:
  - finance
  ---
  # Roberta Masked Language Model Trained On Financial Phrasebank Corpus 
  
  
  This is a Masked Language Model trained with [Roberta](https://huggingface.co/transformers/model_doc/roberta.html) on a Financial Phrasebank Corpus.
  The model is built using Huggingface transformers.
  The model can be found at :[Financial_Roberta](https://huggingface.co/abhilash1910/financial_roberta)
  
  
  ## Specifications
  
  
  The corpus for training is taken from the Financial Phrasebank (Malo et al)[https://www.researchgate.net/publication/251231107_Good_Debt_or_Bad_Debt_Detecting_Semantic_Orientations_in_Economic_Texts]. 
  
  
  ## Model Specification
  
  
  The model chosen for training is [Roberta](https://arxiv.org/abs/1907.11692) with the following specifications:
   1. vocab_size=56000
   2. max_position_embeddings=514
   3. num_attention_heads=12
   4. num_hidden_layers=6
   5. type_vocab_size=1
  
  
  This is trained by using  RobertaConfig from transformers package.
  The model is trained for 10 epochs with a gpu batch size of 64 units. 
  
  
  
  ## Usage Specifications
  
  
  For using this model, we have to first import AutoTokenizer and AutoModelWithLMHead Modules from transformers
  After that we have to specify, the pre-trained model,which in this case is 'abhilash1910/financial_roberta' for the tokenizers and the model.
  
  
  ```python
  from transformers import AutoTokenizer, AutoModelWithLMHead
  
  tokenizer = AutoTokenizer.from_pretrained("abhilash1910/financial_roberta")
  
  model = AutoModelWithLMHead.from_pretrained("abhilash1910/financial_roberta")
  ```
  
  
  After this the model will be downloaded, it will take some time to download all the model files.
  For testing the model, we have to import  pipeline module from transformers and create a masked output model for inference as follows:
  
  
  ```python
  from transformers import pipeline
  model_mask = pipeline('fill-mask', model='abhilash1910/inancial_roberta')
  model_mask("The  company had a <mask> of 20% in 2020.")
  ```
  
  
  Some of the examples are also provided with generic financial statements:
  
  Example 1:
  
  
  ```python
  model_mask("The  company had a <mask> of 20% in 2020.")
  ```
  
  
  Output:
  
  
  ```bash
  [{'sequence': '<s>The  company had a profit of 20% in 2020.</s>',
    'score': 0.023112965747714043,
    'token': 421,
    'token_str': 'Ġprofit'},
   {'sequence': '<s>The  company had a loss of 20% in 2020.</s>',
    'score': 0.021379893645644188,
    'token': 616,
    'token_str': 'Ġloss'},
   {'sequence': '<s>The  company had a year of 20% in 2020.</s>',
    'score': 0.0185744296759367,
    'token': 443,
    'token_str': 'Ġyear'},
   {'sequence': '<s>The  company had a sales of 20% in 2020.</s>',
    'score': 0.018143286928534508,
    'token': 428,
    'token_str': 'Ġsales'},
   {'sequence': '<s>The  company had a value of 20% in 2020.</s>',
    'score': 0.015319528989493847,
    'token': 776,
    'token_str': 'Ġvalue'}]
    ```
   
   Example 2:
   
  ```python
   model_mask("The <mask>  is listed under NYSE")
  ```
  
  Output:
  
  ```bash
  [{'sequence': '<s>The company  is listed under NYSE</s>',
    'score': 0.1566661298274994,
    'token': 359,
    'token_str': 'Ġcompany'},
   {'sequence': '<s>The total  is listed under NYSE</s>',
    'score': 0.05542507395148277,
    'token': 522,
    'token_str': 'Ġtotal'},
   {'sequence': '<s>The value  is listed under NYSE</s>',
    'score': 0.04729423299431801,
    'token': 776,
    'token_str': 'Ġvalue'},
   {'sequence': '<s>The order  is listed under NYSE</s>',
    'score': 0.02533523552119732,
    'token': 798,
    'token_str': 'Ġorder'},
   {'sequence': '<s>The contract  is listed under NYSE</s>',
    'score': 0.02087237872183323,
    'token': 635,
    'token_str': 'Ġcontract'}]
    ```
    
  
  ## Resources
  
  For all resources , please look into the [HuggingFace](https://huggingface.co/) Site and the [Repositories](https://github.com/huggingface).
  