On certain tasks, with the size of LLMs increasing, the performance begins to decrease, such as Redefine-math: tests
whether language models are able to work with common symbols when they are redefined to mean something else; Into-
the-unknown: requires the model to choose which piece of information would help answer a question; Memo-trap: asks
an LM to write a phrase in a way that starts like a famous quote but ends differently6. This is also called Inverse Scaling
Phenomenon. Another interesting phenomenon observed in the scaling of LLMs is called the U-shaped Phenomenon [114].
As the name implies, This phenomenon refers to that as LLM size increases, their performance on certain tasks initially
improves but then starts to decline before eventually improving again, such as on: Hindsight-neglect: it tests whether
language models are able to assess whether a bet was worth taking based on its expected value; NegationQA: this task
Harnessing the Power of LLMs in Practice- A Survey on ChatGPT and Beyond