A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios
<|reference_start|>A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios: We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets.<|reference_end|>
@article{ackerman2024a,
    title={A Novel Metric for Measuring the Robustness of Large Language Models in
  Non-adversarial Scenarios},
    author={Samuel Ackerman, Ella Rabinovich, Eitan Farchi, Ateret Anaby-Tavor},
    journal={arXiv preprint arXiv:2408.01963},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.01963},
    primaryClass={cs.CL stat.AP}
}