ChatGPT produces made-up nonexistent references
This is called &quot;hallucination&quot; and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you&#x27;re never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior?<p>I started looking and it turns out that OpenAI is keenly aware of the hallucination problem, and even wrote about how they&#x27;re correcting for it in this blog post about InstructGPT: <a href="https:&#x2F;&#x2F;openai.com&#x2F;blog&#x2F;instruction-following&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;blog&#x2F;instruction-following&#x2F;</a><p>To summarize: it seems OpenAI is addressing this by taking human-labeled output data from GPT and feeding this into new models, using a reward function to train the future models to behave with less hallucination. This way of working struck me as a mix of software engineering and crop breeding.<p>They discuss the trade-offs involved with doing this. The &quot;hallucination&quot; effect is probably one of the features that make GPT so creative.