Third, a shift from a unified to a challenging setting in evaluating machine learning models.
While unified settings involve a test set with no preference for any specific task, challenging
settings create test sets for specific tasks. Tools like DeepTest [179] use seeds to generate input
transformations for testing, CheckList [156] builds test sets based on templates, and AdaFilter
[148] adversarially constructs tests. However, it is worth noting that AdaFilter may not be entirely
fair as it relies on adversarial examples. HELM [107] evaluates LLMs from different aspects, while
the Big-Bench [172] platform is used to design hard tasks for machine learning models to tackle.
PromptBench [249] aims to evaluate the adversarial robustness of LLMs by creating adversarial
prompts, which is more challenging and the results demonstrated that current LLMs are not robust
to adversarial prompts.
ASurveyonEvaluationofLargeLanguageModels