methods in large-scale language models [111, 129]. One such study evaluates the few-shot learning performance of
GPT-3 on text classification and information extraction tasks [129]. and reveal that the examined LLMs are susceptible
to majority label bias and position bias, where they tend to predict answers based on the frequency or position of the
answers in the training data. Moreover, these LLMs exhibit common token bias, favoring answers that are prevalent in
their pre-training corpus. Recent studies show that this positional bias can be mitigated by selecting proper prompts
[68]. In summary, while LLMs significantly reduce the shortcut learning problem prevalent in fine-tuned models, they
still exhibit some shortcut learning issues and should be approached with caution when deploying them in downstream
applications.
Harnessing the Power of LLMs in Practice- A Survey on ChatGPT and Beyond