I asked GPT-NeoX-20B a hundred arithmetic questions
What? You think this is poor performance?This totally blows my mind. I would never have guessed that GPT could get ANY of these right.I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely...And those multiplications. It's consistently getting the number of digits right and the first two numbers correct. How the hell does this happen?Sure, it's sometimes way off. But generally it is in the right ballpark.I certainly think people should look into what's happening inside the model.