Investigating the Robustness of LLMs on Math Word Problems
<|reference_start|>Investigating the Robustness of LLMs on Math Word Problems: Large Language Models (LLMs) excel at various tasks, including solving math word problems (MWPs), but struggle with real-world problems containing irrelevant information. To address this, we propose a prompting framework that generates adversarial variants of MWPs by adding irrelevant variables. We introduce a dataset, ProbleMATHIC, containing both adversarial and non-adversarial MWPs. Our experiments reveal that LLMs are susceptible to distraction by numerical noise, resulting in an average relative performance drop of ~26% on adversarial MWPs. To mitigate this, we fine-tune LLMs (Llama-2, Mistral) on the adversarial samples from our dataset. Fine-tuning on adversarial training instances improves performance on adversarial MWPs by ~8%, indicating increased robustness to noise and better ability to identify relevant data for reasoning. Finally, to assess the generalizability of our prompting framework, we introduce GSM-8K-Adv, an adversarial variant of the GSM-8K benchmark. LLMs continue to struggle when faced with adversarial information, reducing performance by up to ~6%.<|reference_end|>
@article{anantheswaran2024cutting,
    title={Cutting Through the Noise: Boosting LLM Performance on Math Word
  Problems},
    author={Ujjwala Anantheswaran and Himanshu Gupta and Kevin Scaria and Shreyas
  Verma and Chitta Baral and Swaroop Mishra},
    journal={arXiv preprint arXiv:2406.15444},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.15444},
    primaryClass={cs.CL}
}