<p>ChatGPT does not actually know anything. But more importantly even, it does not know this fact! Hence, it does not know that it does not know.</p>
<p>It is only good at combining text.</p>

CC BY-SA 4.0
2022-12-14T12:38:32.533Z
Note that this is just a philosophical point of view. Who decides that people also know anything? What do you mean by "knowing"? Maybe we also don't know anything. We have memories, which we recall with some algorithm, based on some context. You need to define "to know". Because, if "to know" means "have knowledge of something" and "knowledge" means "info or facts", then ChatGPT certainly knows something, because it can give you some facts. Either it's aware of what it says, it's another question, which also depends on what you mean by "awareness".
nbro
CC BY-SA 4.0
2022-12-14T12:48:05.950Z
So, what one can say for sure is that ChatGPT may give you some facts and correct info, but these facts are just the "most likely" outputs given the data and the inductive bias, i.e. it's a statistical model, and thus it's limited by how the model is defined, the data and the training algorithm. So, it can also give you completely wrong info. The way it produces some output $x$ is, we can say, not equal to how we produce the same output and, most likely, not even close to how we produce it.
nbro
CC BY-SA 4.0
2022-12-15T19:09:13.683Z
There is [a great article](https://journal.sjdm.org/15/15923a/jdm15923a.html) describing this phenomena (which it calls "pseudo-profound bullshit") when done by humans. Cult leaders and snake-oil salesmen are great at stringing together buzzwords into grammatically correct but meaningless sentences, which the unaware listener might think to hold some deep knowledge.
vsz
CC BY-SA 4.0
2022-12-15T20:58:45.997Z
@nbro the aspect of awareness is relevant, I agree. One way to naively put it, is when there is knowledge/awareness regarding the fact, that a specific knowledge is not available.

Humans, in the best case, are aware of their individual lack of knowledge. We have some sort of table of content about things we know. We also know that we may have heard something, but never thought it through ourselves.

ChatGPT and other AI models store and recombine/associate.
DrCommando
CC BY-SA 4.0
2022-12-16T08:59:51.483Z
So interesting comments. Knowledge about facts, and knowledge about how to check new facts against our knowledge base. How to expand our table of contents through valid reasoning.
Jaume Oliver Lafont
CC BY-SA 4.0
2022-12-12T13:23:30.503Z
2022-12-12T13:23:30.503Z
DrCommando
<p>ChatGPT and other GPT-based machine learning models don't actually know anything in the sense you're thinking of.  ChatGPT is a distant descendant of <a href="https://en.wikipedia.org/wiki/Markov_chain" rel="noreferrer">Markov chain</a> text generators such as <a href="https://en.wikipedia.org/wiki/Dissociated_press" rel="noreferrer">Dissociated Press</a>, and works by predicting what is most likely to follow a given sequence of text.</p>
<p>So, when you ask it &quot;Does openai know how to play chess?&quot;, what you're actually getting is the answer to the question &quot;What is most likely to follow the phrase 'Does openai know how to play chess?'?&quot;</p>

CC BY-SA 4.0
2023-04-30T04:35:54.943Z
@Shub, the same way it answers questions.  Code isn't fundamentally any different from written language, and ChatGPT tends to make the same sorts of mistakes, producing code that is syntactically correct but semantically nonsensical.
Mark
CC BY-SA 4.0
2022-12-14T00:31:30.803Z
2022-12-14T00:31:30.803Z
Mark
<p>The original question asked about &quot;<strong>an AI</strong>&quot; generally, yet most of the responses here focus on OpenAI's ChatGPT specifically. Seems like the answer would depend on the specific type of AI being used, not limited to just large language model-based chatbots, but considering other types of knowledge representation systems more generally.</p>
<p>Not being an AI expert, I can't speak to this, but here's some good background reading: <a href="https://en.wikipedia.org/wiki/Knowledge_representation_and_reasoning" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Knowledge_representation_and_reasoning</a></p>

CC BY-SA 4.0
2022-12-16T08:47:26.413Z
2022-12-16T08:47:26.413Z
Trutane
<p>I agree with most the former answers here, and apologies I don't have high enough points yet to comment so had to do this as an answer, but I believe due to inaccuracies, lack of fact checking capabilities etc... in the data its trained on up to 2021 currently at the time of writing, fine tuning via it's API has become popular I believe. Although you were mainly asking about ChatGPT itself I felt its a good idea to consider it's API as part of that which is just as easily accessible, I am currently learning how to fine tune it my self for more specific and accurate results.</p>
<p>So in respect of the question: <strong>Shouldn't the AI reject my question on a subject which is not in its known criteria?
It does so for many other subjects.</strong></p>
<p>It can possibly do this if you train / fine tune it another way on the API, but for the latter part, using the normal ChatGPT and ChatGPT plus user interface part of the site, not the API the AI mostly will not reject your question unless it breaches ethical and moral constructs, which I think is also another important aspect to consider when gaining information from ChatGPT.</p>

CC BY-SA 4.0
2023-04-21T13:18:45.343Z
2023-04-21T13:18:45.343Z
jamiecropley
<p>ChatGPT is a large language model. That means it's very good at stringing together words in ways that humans tend to use them. It's able to construct sentences that are grammatically correct and sound natural, for the most part, because it's been trained on language.</p>
<p>Because it's good at stringing together words, it's able to take your prompt and generate words in a grammatically correct way that's similar to what it's seen before.
But that's all that it's doing: generating words and making sure it sounds natural. It doesn't have any built-in fact checking capabilities, and the manual limitations that OpenAI placed can be fairly easily worked around. Someone in the OpenAI Discord server a few days ago shared a screenshot of the question &quot;What mammal lays the largest eggs?&quot; ChatGPT confidently declared that the elephant lays the largest eggs of any mammal.</p>
<p>While much of the information that ChatGPT was trained on is accurate, always keep in mind that it's just stringing together words with no way to check if what it's saying is accurate. Its sources may have been accurate, but just writing in the style of your sources doesn't mean that the results will themselves be true.</p>

CC BY-SA 4.0
2022-12-12T12:47:49.247Z
Exactly. It finds the words that are most likely to follow the prompt, and has no understanding of anything else.
Oliver Mason
CC BY-SA 4.0
2022-12-12T16:32:53.960Z
I know that on previous GPT models, these kinds of questions with no right answer could lead to some amusing answers. More like this: [Baltimore Orioles effect](https://www.aiweirdness.com/baltimore-orioles-effect/) and [Botsplaining](https://www.aiweirdness.com/botsplaining/) and [Galactica: the AI knowledge base that makes stuff up](https://www.aiweirdness.com/galactica/) and when humans do it [Comparative Illusion](https://en.wikipedia.org/wiki/Comparative_illusion)
Criticize SE actions means ban
CC BY-SA 4.0
2022-12-12T22:40:03.033Z
It's fairly easy to confuse ChatGPT once you understand this principle. For example, if you ask it to describe the Avatar movies it'll confuse James Cameron's Avatar with Avatar: The Last Airbender. Or if you ask it to describe how the planets follow probabilistic orbitals it'll happily confuse Quantum Mechanics and Newtonian gravity for you. All it takes is two concepts that use the same set of words, because the language model has no deeper understanding.
David
CC BY-SA 4.0
2022-12-14T09:32:00.480Z
I'd like to point out that being a "large language model" does not imply "it's very good at stringing together words in ways that humans tend to use them", so I'd rephrase your first sentence, as it implies that every large language model is good at doing that, which may not be the case. What if it's not trained or does not have the correct inductive bias or was trained with "bad" or little data? These are all factors that affect the quality of the LM. Given the attention that this answer and post got, you can be more precise, although some people understand that one does not imply the other
nbro
CC BY-SA 4.0
2022-12-14T12:36:59.323Z
Moreover, you claim "making sure it sounds natural". Right now, I am not familiar with the details of the ChatGPT, but I will soon be, but does it really **ensure** it sounds natural? If it's a statistical model with no constraints on "naturalness", then I doubt that claim is true.
nbro
CC BY-SA 4.0
2022-12-14T13:50:59.123Z
This is why it's less impressive than it looks. Still impressive, sure, but more showy than anything else.
OpenAI was the last straw
CC BY-SA 4.0
2022-12-18T12:37:52.903Z
@David : thanks for the hint about concepts using the same words for different meanings. I asked some questions which mix the concept of entropy from thermodynamics with the concept of entropy from information theory, and the results were hilarious. But the wording was so natural and so confident, that it could easily fool someone who has only a very superficial knowledge about the topic. I fear ChatGPT will be used by pseudoscientists a lot, because it can produce very scientific-sounding articles which seem to have a deeper meaning even when they don't.
vsz
CC BY-SA 4.0
2023-03-15T17:15:56.267Z
Seems [that know it generates the correct answer](https://i.sstatic.net/M9Qge.png) for that specific question. It's learning.
Braiam
CC BY-SA 4.0
2023-06-06T05:44:53.990Z
@Shub You can think of it as LLM generating words letter by letter (what are words anyway but a sequence of letters?) Internally, LLM uses subwords (which may be individual letters) stringed together to form a text output. A code is a text, so it can generate it.
justhalf
CC BY-SA 4.0
2022-12-12T05:41:25.137Z
2022-12-12T05:41:25.137Z
Mithical
<p>After seeing <a href="https://meta.stackoverflow.com/q/421831">StackOverflow's banning of ChatGPT</a>, I explored it out of curiosity. It's marvellous as it can write code by itself!</p>
<p>Later to check if it knows chess as well like Google-Deepmind's AlphaZero AI, I asked below questions:</p>
<blockquote>
<p><strong>Me</strong>: Does openai know how to play chess?<br />
<strong>OpenAI</strong>: It is not clear whether OpenAI knows how to play chess. As an artificial intelligence research organization, OpenAI has developed a number of advanced machine learning algorithms and techniques that allow AI systems to learn and adapt to new tasks and situations. ...</p>
</blockquote>
<blockquote>
<p><strong>Me</strong>: In a chess if white has a king &amp; bishop and if black has only a king, then who can win with a perfect play?<br />
<strong>OpenAI</strong>: In a chess game where white has a king and a bishop and black has only a king, <strong>it is possible for either player to win with perfect play</strong>. This type of endgame is known as a king and bishop versus king endgame, and it is considered one of the most challenging in chess. ...</p>
</blockquote>
<p>Now this answer is entirely wrong. Because as soon as only a bishop (or a knight) remains apart from the kings, it's instantly <a href="https://chess.stackexchange.com/questions/30561/is-it-possible-to-win-with-a-king-and-one-bishop-vs-a-king">declared as a draw</a>!</p>
<p><strong>Question</strong>: Shouldn't the AI reject my question on a subject which is not in its known criteria?<br />
It does so for many other subjects.</p>
<hr />
<p><strong>Note</strong>: Should we replace <code>bishop = rook</code> then ChatGPT answers exactly the same answer with replacing those pieces. However that happens to be true.</p>

CC BY-SA 4.0
2022-12-13T10:41:55.087Z
I think a good way to think about ChatGPT is that it doesn't know anything about anything, and its goal is to bluff its way through a conversation by guessing what you want to hear. Sometimes, the bluff is easier to call.
Stef
CC BY-SA 4.0
2022-12-13T15:57:49.263Z
"that happens to be true" (in the rook case) - really? The GPT answer appears to be saying that **either** player can win; I'm not a chess expert but I'm struggling to see how black can force a win with just a king?
psmears
CC BY-SA 4.0
2022-12-13T18:11:06.610Z
@psmears, sorry.. A small correction here is that, if i replace "bishop" with "rook" then the chatgpt correctly says that white *can* win. No mention of Black's win.
iammilind
CC BY-SA 4.0
2022-12-13T18:45:08.507Z
Related: The [Chinese room](https://en.wikipedia.org/wiki/Chinese_room) argument
Ouroborus
CC BY-SA 4.0
2022-12-14T04:37:10.927Z
"*that happens to be true*" you might want to experiment by retrying the same prompt multiple times and observe that the response may change or even contradict previous response.
Andrew T.
<natural-language-processing><chat-bots><question-answering><board-games><chatgpt>
How does an AI like ChatGPT answer a question in a subject which it may not know?
iammilind
# How does an AI like ChatGPT answer a question in a subject which it may not know?

(Asked by: username_0 on 2022-12-12)

After seeing [StackOverflow's banning of ChatGPT](https://meta.stackoverflow.com/q/421831), I explored it out of curiosity. It's marvellous as it can write code by itself!

Later to check if it knows chess as well like Google-Deepmind's AlphaZero AI, I asked below questions:

> **Me**: Does openai know how to play chess?  
> **OpenAI**: It is not clear whether OpenAI knows how to play chess. As an artificial intelligence research organization, OpenAI has developed a number of advanced machine learning algorithms and techniques that allow AI systems to learn and adapt to new tasks and situations. ...

> **Me**: In a chess if white has a king & bishop and if black has only a king, then who can win with a perfect play?  
> **OpenAI**: In a chess game where white has a king and a bishop and black has only a king, **it is possible for either player to win with perfect play**. This type of endgame is known as a king and bishop versus king endgame, and it is considered one of the most challenging in chess. ...

Now this answer is entirely wrong. Because as soon as only a bishop (or a knight) remains apart from the kings, it's instantly [declared as a draw](https://chess.stackexchange.com/questions/30561/is-it-possible-to-win-with-a-king-and-one-bishop-vs-a-king)!

**Question**: Shouldn't the AI reject my question on a subject which is not in its known criteria?  
It does so for many other subjects.

---

**Note**: Should we replace `bishop = rook` then ChatGPT answers exactly the same answer with replacing those pieces. However that happens to be true.

**Tags:** natural-language-processing, chat-bots, question-answering, board-games, chatgpt

> **Comments:**
> **username_1**: I think a good way to think about ChatGPT is that it doesn't know anything about anything, and its goal is to bluff its way through a conversation by guessing what you want to hear. Sometimes, the bluff is easier to call.  
> **username_2**: "that happens to be true" (in the rook case) - really? The GPT answer appears to be saying that **either** player can win; I'm not a chess expert but I'm struggling to see how black can force a win with just a king?  
> **username_0**: @username_2, sorry.. A small correction here is that, if i replace "bishop" with "rook" then the chatgpt correctly says that white *can* win. No mention of Black's win.  
> **username_3**: Related: The [Chinese room](https://en.wikipedia.org/wiki/Chinese_room) argument  
> **username_4**: "*that happens to be true*" you might want to experiment by retrying the same prompt multiple times and observe that the response may change or even contradict previous response.  

---

## 5 answers

### Answer by username_12 on 2022-12-12. Score: 72 | Accepted answer

ChatGPT is a large language model. That means it's very good at stringing together words in ways that humans tend to use them. It's able to construct sentences that are grammatically correct and sound natural, for the most part, because it's been trained on language.

Because it's good at stringing together words, it's able to take your prompt and generate words in a grammatically correct way that's similar to what it's seen before.
But that's all that it's doing: generating words and making sure it sounds natural. It doesn't have any built-in fact checking capabilities, and the manual limitations that OpenAI placed can be fairly easily worked around. Someone in the OpenAI Discord server a few days ago shared a screenshot of the question "What mammal lays the largest eggs?" ChatGPT confidently declared that the elephant lays the largest eggs of any mammal.

While much of the information that ChatGPT was trained on is accurate, always keep in mind that it's just stringing together words with no way to check if what it's saying is accurate. Its sources may have been accurate, but just writing in the style of your sources doesn't mean that the results will themselves be true.

> **Comments:**
> **username_13**: Exactly. It finds the words that are most likely to follow the prompt, and has no understanding of anything else.  
> **username_14**: I know that on previous GPT models, these kinds of questions with no right answer could lead to some amusing answers. More like this: [Baltimore Orioles effect](https://www.aiweirdness.com/baltimore-orioles-effect/) and [Botsplaining](https://www.aiweirdness.com/botsplaining/) and [Galactica: the AI knowledge base that makes stuff up](https://www.aiweirdness.com/galactica/) and when humans do it [Comparative Illusion](https://en.wikipedia.org/wiki/Comparative_illusion)  
> **username_15**: It's fairly easy to confuse ChatGPT once you understand this principle. For example, if you ask it to describe the Avatar movies it'll confuse James Cameron's Avatar with Avatar: The Last Airbender. Or if you ask it to describe how the planets follow probabilistic orbitals it'll happily confuse Quantum Mechanics and Newtonian gravity for you. All it takes is two concepts that use the same set of words, because the language model has no deeper understanding.  
> **username_6**: I'd like to point out that being a "large language model" does not imply "it's very good at stringing together words in ways that humans tend to use them", so I'd rephrase your first sentence, as it implies that every large language model is good at doing that, which may not be the case. What if it's not trained or does not have the correct inductive bias or was trained with "bad" or little data? These are all factors that affect the quality of the LM. Given the attention that this answer and post got, you can be more precise, although some people understand that one does not imply the other  
> **username_6**: Moreover, you claim "making sure it sounds natural". Right now, I am not familiar with the details of the ChatGPT, but I will soon be, but does it really **ensure** it sounds natural? If it's a statistical model with no constraints on "naturalness", then I doubt that claim is true.  
> **username_16**: This is why it's less impressive than it looks. Still impressive, sure, but more showy than anything else.  
> **username_7**: @username_15 : thanks for the hint about concepts using the same words for different meanings. I asked some questions which mix the concept of entropy from thermodynamics with the concept of entropy from information theory, and the results were hilarious. But the wording was so natural and so confident, that it could easily fool someone who has only a very superficial knowledge about the topic. I fear ChatGPT will be used by pseudoscientists a lot, because it can produce very scientific-sounding articles which seem to have a deeper meaning even when they don't.  
> **username_17**: Seems [that know it generates the correct answer](https://i.sstatic.net/M9Qge.png) for that specific question. It's learning.  
> **username_18**: @Shub You can think of it as LLM generating words letter by letter (what are words anyway but a sequence of letters?) Internally, LLM uses subwords (which may be individual letters) stringed together to form a text output. A code is a text, so it can generate it.  

---

### Answer by username_5 on 2022-12-12. Score: 20

ChatGPT does not actually know anything. But more importantly even, it does not know this fact! Hence, it does not know that it does not know.

It is only good at combining text.

> **Comments:**
> **username_6**: Note that this is just a philosophical point of view. Who decides that people also know anything? What do you mean by "knowing"? Maybe we also don't know anything. We have memories, which we recall with some algorithm, based on some context. You need to define "to know". Because, if "to know" means "have knowledge of something" and "knowledge" means "info or facts", then ChatGPT certainly knows something, because it can give you some facts. Either it's aware of what it says, it's another question, which also depends on what you mean by "awareness".  
> **username_6**: So, what one can say for sure is that ChatGPT may give you some facts and correct info, but these facts are just the "most likely" outputs given the data and the inductive bias, i.e. it's a statistical model, and thus it's limited by how the model is defined, the data and the training algorithm. So, it can also give you completely wrong info. The way it produces some output $x$ is, we can say, not equal to how we produce the same output and, most likely, not even close to how we produce it.  
> **username_7**: There is [a great article](https://journal.sjdm.org/15/15923a/jdm15923a.html) describing this phenomena (which it calls "pseudo-profound bullshit") when done by humans. Cult leaders and snake-oil salesmen are great at stringing together buzzwords into grammatically correct but meaningless sentences, which the unaware listener might think to hold some deep knowledge.  
> **username_5**: @username_6 the aspect of awareness is relevant, I agree. One way to naively put it, is when there is knowledge/awareness regarding the fact, that a specific knowledge is not available.  Humans, in the best case, are aware of their individual lack of knowledge. We have some sort of table of content about things we know. We also know that we may have heard something, but never thought it through ourselves.  ChatGPT and other AI models store and recombine/associate.  
> **username_8**: So interesting comments. Knowledge about facts, and knowledge about how to check new facts against our knowledge base. How to expand our table of contents through valid reasoning.  

---

### Answer by username_9 on 2022-12-14. Score: 15

ChatGPT and other GPT-based machine learning models don't actually know anything in the sense you're thinking of. ChatGPT is a distant descendant of [username_9ov chain](https://en.wikipedia.org/wiki/username_9ov_chain) text generators such as [Dissociated Press](https://en.wikipedia.org/wiki/Dissociated_press), and works by predicting what is most likely to follow a given sequence of text.

So, when you ask it "Does openai know how to play chess?", what you're actually getting is the answer to the question "What is most likely to follow the phrase 'Does openai know how to play chess?'?"

> **Comments:**
> **username_9**: @Shub, the same way it answers questions.  Code isn't fundamentally any different from written language, and ChatGPT tends to make the same sorts of mistakes, producing code that is syntactically correct but semantically nonsensical.  

---

### Answer by username_10 on 2022-12-16. Score: 2

The original question asked about "**an AI**" generally, yet most of the responses here focus on OpenAI's ChatGPT specifically. Seems like the answer would depend on the specific type of AI being used, not limited to just large language model-based chatbots, but considering other types of knowledge representation systems more generally.

Not being an AI expert, I can't speak to this, but here's some good background reading: <https://en.wikipedia.org/wiki/Knowledge_representation_and_reasoning>


---

### Answer by username_11 on 2023-04-21. Score: 0

I agree with most the former answers here, and apologies I don't have high enough points yet to comment so had to do this as an answer, but I believe due to inaccuracies, lack of fact checking capabilities etc... in the data its trained on up to 2021 currently at the time of writing, fine tuning via it's API has become popular I believe. Although you were mainly asking about ChatGPT itself I felt its a good idea to consider it's API as part of that which is just as easily accessible, I am currently learning how to fine tune it my self for more specific and accurate results.

So in respect of the question: **Shouldn't the AI reject my question on a subject which is not in its known criteria?
It does so for many other subjects.**

It can possibly do this if you train / fine tune it another way on the API, but for the latter part, using the normal ChatGPT and ChatGPT plus user interface part of the site, not the API the AI mostly will not reject your question unless it breaches ethical and moral constructs, which I think is also another important aspect to consider when gaining information from ChatGPT.


---

