Pre-Calc: Learning to Use the Calculator Improves Numeracy in Language Models
<|reference_start|>Pre-Calc: Learning to Use the Calculator Improves Numeracy in Language Models: Quantitative and numerical comprehension in language is an important task in many fields like education and finance, but still remains a challenging task for language models. While tool and calculator usage has shown to be helpful to improve mathematical reasoning in large pretrained decoder-only language models, this remains unexplored for smaller language models with encoders. In this paper, we propose Pre-Calc, a simple pre-finetuning objective of learning to use the calculator for both encoder-only and encoder-decoder architectures, formulated as a discriminative and generative task respectively. We pre-train BERT and RoBERTa for discriminative calculator use and Flan-T5 for generative calculator use on the MAWPS, SVAMP, and AsDiv-A datasets, which improves performance on downstream tasks that require numerical understanding. Our code and data are available at https://github.com/calc-cmu/pre-calc.<|reference_end|>
@article{veerendranath2024pre-calc:,
    title={Pre-Calc: Learning to Use the Calculator Improves Numeracy in Language
  Models},
    author={Vishruth Veerendranath, Vishwa Shah, Kshitish Ghate},
    journal={arXiv preprint arXiv:2404.14355},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.14355},
    primaryClass={cs.CL cs.AI}
}