GPT-4 Takes a New Midterm and Gets an A
People have a hard time understanding what zero-shot, generalization, and memorization mean. Generative models are VERY hard to generate even before we begin to look at LLMs. Let me explain and hopefully we can stop this madness.<p>Zero Shot:<p>&gt; Zero-shot learning consists in learning how to recognize new concepts by just having a description of them.[0]<p>Here&#x27;s an example of a zero shot task. Suppose a LLM is trained only on text. Then you fine tune for image classification, but those do not include cats (of any type). Then you ask it to classify a picture of a cat. An object it has NEVER SEEN BEFORE.<p>The community has been pulling a fast one recently. Recent works like Imagen, DALLE, Parti, etc have been claiming a &quot;Zero-Shot MS-COCO&quot;. These are 100% bullshit claims. You can go look at images in the COCO dataset[1] and then search them in the LION dataset[2] (CLIP retrieval). You&#x27;ll see that there are similar images that have the same classes. These models may not have seen the exact same image before, but they&#x27;ve seen plenty of examples. This is NOT zero shot.<p>Generalization:<p>Google&#x27;s developer pages[3] defines generalization as<p>&gt; Generalization refers to your model&#x27;s ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model.<p>The irony being that these &quot;Zero-Shot MS-COCO&quot; results they give are actually good tests for generalization. Assuming the datasets these were trained on were held constant (they aren&#x27;t) then this would be a great comparison.<p>Memorization:<p>This is the &quot;stochastic parrot&quot; stuff. I don&#x27;t have GPT4 but let&#x27;s use chat.openai&#x27;s current 3.5[4]. I asked:<p>&gt; Which weighs more, a kilogram of feathers or two of bricks?<p>&gt; A kilogram of feathers weighs the same as two bricks. Weight is a measure of mass, and one kilogram is one kilogram, regardless of the material. So, a kilogram of feathers and two bricks would have the same weight. However, the volume and size of the two objects would likely be very different, as feathers are much less dense than bricks.<p>Which is an absurd answer. The system memorized the resultant pattern for &quot;Which weights more, a kilogram of feathers or a kilogram of bricks&quot; and responds to any tweaked variation of that as if it was the original version. It is &quot;over-fit&quot;. The answer is even slightly more insane than that, because it didn&#x27;t correctly pick up that &quot;kilogram&quot; is applied to both numbers and just responds as if &quot;two bricks&quot; is &quot;a kilogram of bricks&quot;. This is just pattern matching.<p>This is why some people get really good code answers and others have a really hard time. It depends on what kinds of systems they are coding. Likely strongly correlates with people who view their job closer to &quot;copy paste from stack overflow&quot;. Some researchers tested some memorization[9] with Codeforce and looked at the distribution based on the cutoff date.<p>Back to the convo:<p>Now that we know what we&#x27;re talking about and can have a consistent definition of words, we can talk about these things. GPT is neither a pure memorization machine[5] nor &quot;intelligent&quot;[6][7]. It is a language model, which we are having an incredibly difficult time evaluating. We can&#x27;t have sane conversations about these systems because the hype creates a bimodal distribution of conversations -- oversell, undersell -- and neither are anywhere near accurate. These systems are impressive but we must also be very careful in evaluation.<p>So the content of the blog? As a generative researcher, I&#x27;m not surprised by looking at his questions. The first question has a clear pattern to it that you&#x27;d see in an economic class and the author even shows a simple equation: x - y = alpha * z (x,y,alpha provided, solve for z). The second question (Californians moving to Texas) has been written about and you&#x27;ll find a lot of Google results. So it should be unsurprising that a system trained on a large chunk of the internet can regurgitate a good answer. There&#x27;s 2 things surprising about this though. 1) from a research perspective, it is quite cool that GPT is creating a weak causal (associative) diagram and can write good conclusions on this. There&#x27;s sparks of causal reasoning in GPT and that&#x27;s awesome (See Judea&#x27;s Twitter feed, he&#x27;s been playing around)[8]. 2) That neither GPT nor Dr. Caplan noted how California isn&#x27;t monolithic in political affiliation and that this is actually an unsurprising phenomena if we consider this and could be nuanced at who is moving. It is quite possible that conservative people are moving to Texas because they are they are annoyed by the politics. This has a directly opposite conclusion from what both of them wrote, and has been written about (they are prioritizing politics). But I&#x27;ll give both a pass because the question is slightly ambiguous in that it is unclear if &quot;Californians who are liberal&quot; are moving or &quot;People from California, which is a liberal state&quot; (does &quot;liberal&quot; apply to the state or the person?).<p>So there&#x27;s no zero-shot here. &quot;New&quot; doesn&#x27;t mean &quot;novel&quot;. But why would a midterm be novel? Great way to fuck over your students. Honestly, if GPT couldn&#x27;t pass the midterm I&#x27;d be surprised. But I can tell you how to make GPT fail. Use more math. It still has issues with that. But at the same time I wouldn&#x27;t be too surprised if it could pass the Physics or Chemistry GREs even if it &quot;never saw it before.&quot; Just scraping Reddit would be enough to do decently well. The Math GRE would be impressive though, but only because it is a weak point. There&#x27;s more than enough info for it to memorize. These tests do not measure intelligence nor even how good of a scientist&#x2F;researcher you are. They test your ability to memorize and pattern match under stressful conditions. Take the conclusions lightly.<p>Okay, now that we got that settled, I&#x27;m signing back off. Too much to do and you all with the hype are making it harder. The internet makes me too frustrated lately. I just want to build ML systems and it is hard to do with all these strong opinions with low expertise taking center stage. Can we stop with these blogs? They aren&#x27;t helping. The real danger we&#x27;re facing with A{G,}I is that we can&#x27;t even have honest conversations about the danger that these systems do pose. Overselling the danger is just as bad as underselling it. Being an armchair expert isn&#x27;t helping by being &quot;good enough&quot; it is harmful, especially when you defend your opinion so strongly. I&#x27;ll tell you the truth, us in the field are still trying to figure all this shit out. If we&#x27;re having a hard time then don&#x27;t trust your friend who has just a handful of ML projects. Even a few papers may not be a good enough signal. The system is noisy, lower your trust.<p>TDLR: be careful with hyped subjects. You&#x27;re not getting an accurate picture and many people aren&#x27;t acting in good faith.<p>[0] <a href="https:&#x2F;&#x2F;proceedings.mlr.press&#x2F;v37&#x2F;romera-paredes15.html" rel="nofollow">https:&#x2F;&#x2F;proceedings.mlr.press&#x2F;v37&#x2F;romera-paredes15.html</a><p>[1] <a href="https:&#x2F;&#x2F;cocodataset.org&#x2F;#explore" rel="nofollow">https:&#x2F;&#x2F;cocodataset.org&#x2F;#explore</a><p>[2] <a href="https:&#x2F;&#x2F;rom1504.github.io&#x2F;clip-retrieval&#x2F;" rel="nofollow">https:&#x2F;&#x2F;rom1504.github.io&#x2F;clip-retrieval&#x2F;</a><p>[3] <a href="https:&#x2F;&#x2F;developers.google.com&#x2F;machine-learning&#x2F;crash-course&#x2F;generalization&#x2F;video-lecture" rel="nofollow">https:&#x2F;&#x2F;developers.google.com&#x2F;machine-learning&#x2F;crash-course&#x2F;...</a><p>[4] <a href="https:&#x2F;&#x2F;chat.openai.com&#x2F;chat" rel="nofollow">https:&#x2F;&#x2F;chat.openai.com&#x2F;chat</a><p>[5] <a href="https:&#x2F;&#x2F;www.newyorker.com&#x2F;tech&#x2F;annals-of-technology&#x2F;chatgpt-is-a-blurry-jpeg-of-the-web" rel="nofollow">https:&#x2F;&#x2F;www.newyorker.com&#x2F;tech&#x2F;annals-of-technology&#x2F;chatgpt-...</a><p>[6] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2303.12712" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2303.12712</a><p>[7] Intelligence is hard to define and we won&#x27;t try to here, so the quotes. (Is an ant intelligent? I would say &quot;yes&quot;, but I also understand a &quot;no&quot; answer) But the point is that people are over-selling the intelligence.<p>[8] <a href="https:&#x2F;&#x2F;twitter.com&#x2F;yudapearl" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;yudapearl</a><p>[9] (Twitter now marks this as an unsafe website. Good going Elon...) <a href="https:&#x2F;&#x2F;aisnakeoil.substack.com&#x2F;p&#x2F;gpt-4-and-professional-benchmarks" rel="nofollow">https:&#x2F;&#x2F;aisnakeoil.substack.com&#x2F;p&#x2F;gpt-4-and-professional-ben...</a>