readme: ---
  language: en
  license: mit
  datasets:
  - arxmliv
  - math-stackexchange
  ---
  
  # MathBERTa model
  
  Pretrained model on English language and LaTeX using a masked language modeling
  (MLM) objective. It was introduced in [this paper][1] and first released in
  [this repository][2]. This model is case-sensitive: it makes a difference
  between english and English.
  
   [1]: http://ceur-ws.org/Vol-3180/paper-06.pdf
   [2]: https://github.com/witiko/scm-at-arqmath3
  
  ## Model description
  
  MathBERTa is [the RoBERTa base transformer model][3] whose [tokenizer has been
  extended with LaTeX math symbols][7] and which has been [fine-tuned on a large
  corpus of English mathematical texts][8].
  
  Like RoBERTa, MathBERTa has been fine-tuned with the Masked language modeling
  (MLM) objective. Taking a sentence, the model randomly masks 15% of the words
  and math symbols in the input then run the entire masked sentence through the
  model and has to predict the masked words and symbols. This way, the model
  learns an inner representation of the English language and LaTeX that can then
  be used to extract features useful for downstream tasks.
  
   [3]: https://huggingface.co/roberta-base
   [7]: https://github.com/Witiko/scm-at-arqmath3/blob/main/02-train-tokenizers.ipynb
   [8]: https://github.com/witiko/scm-at-arqmath3/blob/main/03-finetune-roberta.ipynb
  
  ## Intended uses & limitations
  
  You can use the raw model for masked language modeling, but it's mostly
  intended to be fine-tuned on a downstream task.  See the [model
  hub][4] to look for fine-tuned versions on a task that interests you.
  
  Note that this model is primarily aimed at being fine-tuned on tasks that use
  the whole sentence (potentially masked) to make decisions, such as sequence
  classification, token classification or question answering. For tasks such as
  text generation you should look at model like GPT2.
  
   [4]: https://huggingface.co/models?filter=roberta
  
  ### How to use
  
  
  *Due to the large number of added LaTeX tokens, MathBERTa is affected by [a
  software bug in the 🤗 Transformers library][9] that causes it to load for tens
  of minutes. The bug was [fixed in version 4.20.0][10].*
  
  You can use this model directly with a pipeline for masked language modeling:
  
  ```python
  >>> from transformers import pipeline
  >>> unmasker = pipeline('fill-mask', model='witiko/mathberta')
  >>> unmasker(r"If [MATH] \theta = \pi [/MATH] , then [MATH] \sin(\theta) [/MATH] is <mask>.")
  
  [{'sequence': ' If \\theta = \\pi, then\\sin(\\theta ) is zero.'
    'score': 0.23291291296482086,
    'token': 4276,
    'token_str': ' zero'},
   {'sequence': ' If \\theta = \\pi, then\\sin(\\theta ) is 0.'
    'score': 0.11734672635793686,
    'token': 321,
    'token_str': ' 0'},
   {'sequence': ' If \\theta = \\pi, then\\sin(\\theta ) is real.'
    'score': 0.0793389230966568,
    'token': 588,
    'token_str': ' real'},
   {'sequence': ' If \\theta = \\pi, then\\sin(\\theta ) is 1.'
    'score': 0.0753420740365982,
    'token': 112,
    'token_str': ' 1'},
   {'sequence': ' If \\theta = \\pi, then\\sin(\\theta ) is even.'
    'score': 0.06487451493740082,
    'token': 190,
    'token_str': ' even'}]
  ```
  
  Here is how to use this model to get the features of a given text in PyTorch:
  
  ```python
  from transformers import AutoTokenizer, AutoModel
  tokenizer = AutoTokenizer.from_pretrained('witiko/mathberta')
  model = AutoModel.from_pretrained('witiko/mathberta')
  text = r"Replace me by any text and [MATH] \text{math} [/MATH] you'd like."
  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)
  ```
  
  ## Training data
  
  Our model was fine-tuned on two datasets:
  
  - [ArXMLiv 2020][5], a dataset consisting of 1,581,037 ArXiv documents.
  - [Math StackExchange][6], a dataset of  2,466,080 questions and answers.
  
  Together theses datasets weight 52GB of text and LaTeX.
  
  ## Intrinsic evaluation results
  
  Our model achieves the following intrinsic evaluation results:
  
   ![Intrinsic evaluation results of MathBERTa][11]
  
   [5]: https://sigmathling.kwarc.info/resources/arxmliv-dataset-2020/
   [6]: https://www.cs.rit.edu/~dprl/ARQMath/arqmath-resources.html
   [9]: https://github.com/huggingface/transformers/issues/16936
   [10]: https://github.com/huggingface/transformers/pull/17119
   [11]: https://huggingface.co/witiko/mathberta/resolve/main/learning-curves.png
  
  ## Citing
  
  ### Text
  
  Vít Novotný and Michal Štefánik. “Combining Sparse and Dense Information
  Retrieval. Soft Vector Space Model and MathBERTa at ARQMath-3”.
  In: *Proceedings of the Working Notes of CLEF 2022*. To Appear.
  CEUR-WS, 2022.
  
  ### Bib(La)TeX
  
  ``` bib
  @inproceedings{novotny2022combining,
    booktitle = {Proceedings of the Working Notes of {CLEF} 2022},
    editor = {Faggioli, Guglielmo and Ferro, Nicola and Hanbury, Allan and Potthast, Martin},
    issn = {1613-0073},
    title = {Combining Sparse and Dense Information Retrieval},
    subtitle = {Soft Vector Space Model and MathBERTa at ARQMath-3 Task 1 (Answer Retrieval)},
    author = {Novotný, Vít and Štefánik, Michal},
    publisher = {{CEUR-WS}},
    year = {2022