readme: ---
  language: 
    - en
  tags:
  - retrieval
  - math-retrieval
  datasets:
  - MathematicalStackExchange
  - ARQMath
  ---
  
  # ALBERT for ARQMath 3
  
  This repository contains our best model for ARQMath 3, the math_10 model. It was initialised from ALBERT-base-v2 and further pre-trained on Math StackExchange in three different stages. We also added more LaTeX tokens to the tokenizer to enable a better tokenization of mathematical formulas. math_10 was fine-tuned on a classification task to determine whether a given question (sequence 1) matches a given answer (sequence 2). The classification output can be used for ranking the best answers. For further details, please read our paper: http://ceur-ws.org/Vol-3180/paper-07.pdf. 
  
  
  ## Other Models for ARQMath 3
  We plan on also publishing the other fine-tuned models as well as the base models. Links to these repositories will be added here soon.
  
  | Model       | Initialised from | Pre-training               | Fine-Tuned                          | Link |
  |-------------|------------------|----------------------------|-------------------------------------|------|
  | roberta_10  | RoBERTa          | MathSE (1)                 | yes, N=10 MathSE                    |      |
  | base_10     | ALBERT           | MathSE (1)                 | yes, N=10 MathSE                    |      |
  | math_10_add | ALBERT           | MathSE (1)-(3)             | yes, N=10 MathSE and annotated data |      |
  | Khan_SE_10  | ALBERT           | MathSE (1)                 | yes, N=10 MathSE                    |      |
  | roberta     | RoBERTa          | MathSE (1)                 | no                                  | [AnReu/math_pretrained_roberta](https://huggingface.co/AnReu/math_pretrained_roberta)     |
  | math albert | ALBERT           | MathSE (1)-(3)             | no                                  | [AnReu/math_albert](https://huggingface.co/AnReu/math_albert) |
  | base        | ALBERT           | MathSE (1)                 | no                                  |      |
  | Khan_SE     | ALBERT           | MathSE (1) mixed with Khan | no                                  |      |
  
  ### Update
  
  We have also further pre-trained a BERT-base-cased model in the same way as our ALBERT model. You can find it here: [AnReu/math_pretrained_bert](https://huggingface.co/AnReu/math_pretrained_bert).
  
  # Usage
  
  ```python
  # based on https://huggingface.co/docs/transformers/main/en/task_summary#sequence-classification
  from transformers import AutoTokenizer, AutoModelForSequenceClassification
  
  tokenizer = AutoTokenizer.from_pretrained("AnReu/albert-for-arqmath-3")
  
  model = AutoModelForSequenceClassification.from_pretrained("AnReu/albert-for-arqmath-3")
  
  classes = ["non relevant", "relevant"]
  
  sequence_0 = "How can I calculate x in $3x = 5$"
  sequence_1 = "Just divide by 3: $x = \\frac{5}{3}$"
  sequence_2 = "The general rule for squaring a sum is $(a+b)^2=a^2+2ab+b^2$"
  
  # The tokenizer will automatically add any model specific separators (i.e. <CLS> and <SEP>) and tokens to
  # the sequence, as well as compute the attention masks.
  irrelevant = tokenizer(sequence_0, sequence_2, return_tensors="pt")
  relevant = tokenizer(sequence_0, sequence_1, return_tensors="pt")
  
  irrelevant_classification_logits = model(**irrelevant).logits
  relevant_classification_logits = model(**relevant).logits
  
  irrelevant_results = torch.softmax(irrelevant_classification_logits, dim=1).tolist()[0]
  relevant_results = torch.softmax(relevant_classification_logits, dim=1).tolist()[0]
  
  # Should be irrelevant
  for i in range(len(classes)):
      print(f"{classes[i]}: {int(round(irrelevant_results[i] * 100))}%")
  
  # Should be relevant
  for i in range(len(classes)):
      print(f"{classes[i]}: {int(round(relevant_results[i] * 100))}%")
  
  ```
  
  # Citation
  
  If you find this model useful, consider citing our paper:
  ```
  @article{reusch2022transformer,
    title={Transformer-Encoder and Decoder Models for Questions on Math},
    author={Reusch, Anja and Thiele, Maik and Lehner, Wolfgang},
    year={2022},
    organization={CLEF}
  }
  ```