In software engineering tasks, ChatGPT generally performs well and provides detailed responses,
often surpassing both human expert output and SOTA output. However, for certain tasks such as
code vulnerability detection and information retrieval-based test prioritization, the current version
of ChatGPT fails to provide accurate answers, rendering it unsuitable for these specific tasks [171].
In commonsense planning tasks, LLMs may not perform well, even in simple planning tasks
where humans excel [183, 184]. Pallagani et al. [140] demonstrated that the fine-tuned CodeT5
[202] performs the best across all considered domains, with the shortest inference time. Moreover,
it explored the capability of LLMs for plan generalization and found that their generalization
capabilities appear to be limited. It turns out that LLMs can handle simple engineering tasks, but
they perform poorly on complex engineering tasks.
ASurveyonEvaluationofLargeLanguageModels