<p>As a psychologist, Alison Gopnik's analogy of a &quot;fuzzy jpeg&quot; refers to human brain development. Young children's brains might be like a &quot;fuzzy jpeg,&quot; but they can still demonstrate reasonable learning and knowledge. She extended the analogy to artificial intelligence/machine learning. While a neural network is still fairly &quot;fuzzy,&quot; it can still produce reasonable and sometimes surprising results.</p>
<p>Of course, at the risk of overfitting, the more we train a neural network the less fuzzy it becomes.</p>
<p>But I think it is a poor analogy, given that a JPEG is a static image and a neural network is proven to be Turing complete (computationally universal).</p>

CC BY-SA 4.0
2024-04-03T11:47:12.573Z
I think some mixup on "Turing complete". The universal approximation theorem says nothing about that. I believe that recurrent neural networks can be viewed as trainable Turing machines, but not sure if there's proof of universal computation? If you have a reference for the proof you mention at the end, I would appreciate a link
Neil Slater
CC BY-SA 4.0
2024-04-03T19:41:56.253Z
The Universal approximation theorem only addresses continuous functions, not sufficient to be Turing complete. 
I have several references to papers that address neural networks as Turing complete, therefore also computational universal. I have a list of attributions to papers that prove certain classes and configurations of ANNs as Turing complete. I will share the links to the online versions I can find.
Mark_Sagecy
CC BY-SA 4.0
2024-04-03T19:54:36.173Z
Here are a few relevant papers addressing the topic of Turing completeness of ANNs, there are many more. "How Powerful are Decoder-Only Transformer Neural Models?"  https://arxiv.org/pdf/2305.17026.pdf "On the Turing Completeness of Modern Neural Network Architecture" https://arxiv.org/abs/1901.03429 "Attention is Turing Complete" https://jmlr.org/papers/volume22/20-302/20-302.pdf
Mark_Sagecy
CC BY-SA 4.0
2024-04-02T18:55:50.690Z
2024-04-02T18:55:50.690Z
Mark_Sagecy
<p>We should</p>
<blockquote>
<p>conceptualize LLMs as very quirky and very experimental if advanced
information retrieval systems (fuzzy JPEGs)</p>
</blockquote>
<p>according to <strong>point 7</strong> in the screenshot below (<a href="https://twitter.com/pmarca/status/1774547180713631885" rel="nofollow noreferrer">grabbed from pmarca's Twitter feed</a>).</p>
<p>Could someone please help me understand how &quot;fuzzy&quot; and &quot;JPEGs&quot; are relevant here?</p>
<p><a href="https://i.sstatic.net/TZIzT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TZIzT.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2024-04-01T04:35:12.853Z
JPEGs are not a great analogy, but the idea is lossy compression. If you use arithmetic coding on the indices of correct output from the LLM, then minimizing cross-entropy loss minimizes your coding scheme. If you were to always use the most likely output (or only sample from the top 10% of tokens), you'd have a lossy compression.
programjames
CC BY-SA 4.0
2024-04-01T06:09:51.223Z
@programjames here, does "fuzzy" mean "fuzzy logic" or a "not sharp image"?
thanks_in_advance
CC BY-SA 4.0
2024-04-02T00:33:19.250Z
The latter. JPEGs are already lossy, so it wouldn't make sense to add *more* noise when reconstructing them. The hidden layers of LLMs do happen to implement fuzzy logic though, so that is confusing.
programjames
CC BY-SA 4.0
2024-04-02T14:31:06.260Z
For reproducibility, please, provide the link to the tweet
nbro
CC BY-SA 4.0
2024-04-02T14:48:21.187Z
@nbro done. https://twitter.com/pmarca/status/1774547180713631885
thanks_in_advance
CC BY-SA 4.0
2024-04-02T15:08:25.607Z
That sentence makes little sense to me. Possibly, there's a typo. Specifically, the "if" seems to be a typo. I suppose they meant to write "as". I wouldn't spend much time on deciphering tweets, which were maybe written by people that don't know much about the topic. I've never heard of "fuzzy JPEGs". JPEGs are lossy (I think), which means you can lose info. Maybe that's what the "fuzzy" refers to. I'm unaware of any algorithm that has that name, but I know little about those things.
nbro
<large-language-models><fuzzy-logic><information-retrieval>
LLMs as "fuzzy JPEGs"
thanks_in_advance
# LLMs as "fuzzy JPEGs"

(Asked by: username_0 on 2024-03-31)

We should

> conceptualize LLMs as very quirky and very experimental if advanced
> information retrieval systems (fuzzy JPEGs)

according to **point 7** in the screenshot below ([grabbed from pmarca's Twitter feed](https://twitter.com/pmarca/status/1774547180713631885)).

Could someone please help me understand how "fuzzy" and "JPEGs" are relevant here?

[![enter image description here](https://i.sstatic.net/TZIzT.png)](https://i.sstatic.net/TZIzT.png)

**Tags:** large-language-models, fuzzy-logic, information-retrieval

> **Comments:**
> **username_1**: JPEGs are not a great analogy, but the idea is lossy compression. If you use arithmetic coding on the indices of correct output from the LLM, then minimizing cross-entropy loss minimizes your coding scheme. If you were to always use the most likely output (or only sample from the top 10% of tokens), you'd have a lossy compression.  
> **username_0**: @username_1 here, does "fuzzy" mean "fuzzy logic" or a "not sharp image"?  
> **username_1**: The latter. JPEGs are already lossy, so it wouldn't make sense to add *more* noise when reconstructing them. The hidden layers of LLMs do happen to implement fuzzy logic though, so that is confusing.  
> **username_2**: For reproducibility, please, provide the link to the tweet  
> **username_0**: @username_2 done. https://twitter.com/pmarca/status/1774547180713631885  
> **username_2**: That sentence makes little sense to me. Possibly, there's a typo. Specifically, the "if" seems to be a typo. I suppose they meant to write "as". I wouldn't spend much time on deciphering tweets, which were maybe written by people that don't know much about the topic. I've never heard of "fuzzy JPEGs". JPEGs are lossy (I think), which means you can lose info. Maybe that's what the "fuzzy" refers to. I'm unaware of any algorithm that has that name, but I know little about those things.  

---

## 1 answers

### Answer by username_3 on 2024-04-02. Score: 0

As a psychologist, Alison Gopnik's analogy of a "fuzzy jpeg" refers to human brain development. Young children's brains might be like a "fuzzy jpeg," but they can still demonstrate reasonable learning and knowledge. She extended the analogy to artificial intelligence/machine learning. While a neural network is still fairly "fuzzy," it can still produce reasonable and sometimes surprising results.

Of course, at the risk of overfitting, the more we train a neural network the less fuzzy it becomes.

But I think it is a poor analogy, given that a JPEG is a static image and a neural network is proven to be Turing complete (computationally universal).

> **Comments:**
> **username_4**: I think some mixup on "Turing complete". The universal approximation theorem says nothing about that. I believe that recurrent neural networks can be viewed as trainable Turing machines, but not sure if there's proof of universal computation? If you have a reference for the proof you mention at the end, I would appreciate a link  
> **username_3**: The Universal approximation theorem only addresses continuous functions, not sufficient to be Turing complete.  I have several references to papers that address neural networks as Turing complete, therefore also computational universal. I have a list of attributions to papers that prove certain classes and configurations of ANNs as Turing complete. I will share the links to the online versions I can find.  
> **username_3**: Here are a few relevant papers addressing the topic of Turing completeness of ANNs, there are many more. "How Powerful are Decoder-Only Transformer Neural Models?"  https://arxiv.org/pdf/2305.17026.pdf "On the Turing Completeness of Modern Neural Network Architecture" https://arxiv.org/abs/1901.03429 "Attention is Turing Complete" https://jmlr.org/papers/volume22/20-302/20-302.pdf  

---

