Answers.Body: <p>As a psychologist, Alison Gopnik's analogy of a &quot;fuzzy jpeg&quot; refers to human brain development. Young children's brains might be like a &quot;fuzzy jpeg,&quot; but they can still demonstrate reasonable learning and knowledge. She extended the analogy to artificial intelligence/machine learning. While a neural network is still fairly &quot;fuzzy,&quot; it can still produce reasonable and sometimes surprising results.</p>
  <p>Of course, at the risk of overfitting, the more we train a neural network the less fuzzy it becomes.</p>
  <p>But I think it is a poor analogy, given that a JPEG is a static image and a neural network is proven to be Turing complete (computationally universal).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-03T11:47:12.573Z
Answers.Text: I think some mixup on "Turing complete". The universal approximation theorem says nothing about that. I believe that recurrent neural networks can be viewed as trainable Turing machines, but not sure if there's proof of universal computation? If you have a reference for the proof you mention at the end, I would appreciate a link
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-03T19:41:56.253Z
Answers.Text: The Universal approximation theorem only addresses continuous functions, not sufficient to be Turing complete. 
  I have several references to papers that address neural networks as Turing complete, therefore also computational universal. I have a list of attributions to papers that prove certain classes and configurations of ANNs as Turing complete. I will share the links to the online versions I can find.
Answers.UserDisplayName: Mark_Sagecy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-03T19:54:36.173Z
Answers.Text: Here are a few relevant papers addressing the topic of Turing completeness of ANNs, there are many more. "How Powerful are Decoder-Only Transformer Neural Models?"  https://arxiv.org/pdf/2305.17026.pdf "On the Turing Completeness of Modern Neural Network Architecture" https://arxiv.org/abs/1901.03429 "Attention is Turing Complete" https://jmlr.org/papers/volume22/20-302/20-302.pdf
Answers.UserDisplayName: Mark_Sagecy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-02T18:55:50.690Z
Answers.LastActivityDate: 2024-04-02T18:55:50.690Z
Answers.UserDisplayName: Mark_Sagecy
Body: <p>We should</p>
  <blockquote>
  <p>conceptualize LLMs as very quirky and very experimental if advanced
  information retrieval systems (fuzzy JPEGs)</p>
  </blockquote>
  <p>according to <strong>point 7</strong> in the screenshot below (<a href="https://twitter.com/pmarca/status/1774547180713631885" rel="nofollow noreferrer">grabbed from pmarca's Twitter feed</a>).</p>
  <p>Could someone please help me understand how &quot;fuzzy&quot; and &quot;JPEGs&quot; are relevant here?</p>
  <p><a href="https://i.sstatic.net/TZIzT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TZIzT.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-01T04:35:12.853Z
Comments.Text: JPEGs are not a great analogy, but the idea is lossy compression. If you use arithmetic coding on the indices of correct output from the LLM, then minimizing cross-entropy loss minimizes your coding scheme. If you were to always use the most likely output (or only sample from the top 10% of tokens), you'd have a lossy compression.
Comments.UserDisplayName: programjames
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-01T06:09:51.223Z
Comments.Text: @programjames here, does "fuzzy" mean "fuzzy logic" or a "not sharp image"?
Comments.UserDisplayName: thanks_in_advance
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T00:33:19.250Z
Comments.Text: The latter. JPEGs are already lossy, so it wouldn't make sense to add *more* noise when reconstructing them. The hidden layers of LLMs do happen to implement fuzzy logic though, so that is confusing.
Comments.UserDisplayName: programjames
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T14:31:06.260Z
Comments.Text: For reproducibility, please, provide the link to the tweet
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T14:48:21.187Z
Comments.Text: @nbro done. https://twitter.com/pmarca/status/1774547180713631885
Comments.UserDisplayName: thanks_in_advance
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-02T15:08:25.607Z
Comments.Text: That sentence makes little sense to me. Possibly, there's a typo. Specifically, the "if" seems to be a typo. I suppose they meant to write "as". I wouldn't spend much time on deciphering tweets, which were maybe written by people that don't know much about the topic. I've never heard of "fuzzy JPEGs". JPEGs are lossy (I think), which means you can lose info. Maybe that's what the "fuzzy" refers to. I'm unaware of any algorithm that has that name, but I know little about those things.
Comments.UserDisplayName: nbro
Tags: <large-language-models><fuzzy-logic><information-retrieval>
Title: LLMs as "fuzzy JPEGs"
UserDisplayName: thanks_in_advance
ThreadText: # LLMs as "fuzzy JPEGs"
  
  (Asked by: username_0 on 2024-03-31)
  
  We should
  
  > conceptualize LLMs as very quirky and very experimental if advanced
  > information retrieval systems (fuzzy JPEGs)
  
  according to **point 7** in the screenshot below ([grabbed from pmarca's Twitter feed](https://twitter.com/pmarca/status/1774547180713631885)).
  
  Could someone please help me understand how "fuzzy" and "JPEGs" are relevant here?
  
  [![enter image description here](https://i.sstatic.net/TZIzT.png)](https://i.sstatic.net/TZIzT.png)
  
  **Tags:** large-language-models, fuzzy-logic, information-retrieval
  
  > **Comments:**
  > **username_1**: JPEGs are not a great analogy, but the idea is lossy compression. If you use arithmetic coding on the indices of correct output from the LLM, then minimizing cross-entropy loss minimizes your coding scheme. If you were to always use the most likely output (or only sample from the top 10% of tokens), you'd have a lossy compression.  
  > **username_0**: @username_1 here, does "fuzzy" mean "fuzzy logic" or a "not sharp image"?  
  > **username_1**: The latter. JPEGs are already lossy, so it wouldn't make sense to add *more* noise when reconstructing them. The hidden layers of LLMs do happen to implement fuzzy logic though, so that is confusing.  
  > **username_2**: For reproducibility, please, provide the link to the tweet  
  > **username_0**: @username_2 done. https://twitter.com/pmarca/status/1774547180713631885  
  > **username_2**: That sentence makes little sense to me. Possibly, there's a typo. Specifically, the "if" seems to be a typo. I suppose they meant to write "as". I wouldn't spend much time on deciphering tweets, which were maybe written by people that don't know much about the topic. I've never heard of "fuzzy JPEGs". JPEGs are lossy (I think), which means you can lose info. Maybe that's what the "fuzzy" refers to. I'm unaware of any algorithm that has that name, but I know little about those things.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2024-04-02. Score: 0
  
  As a psychologist, Alison Gopnik's analogy of a "fuzzy jpeg" refers to human brain development. Young children's brains might be like a "fuzzy jpeg," but they can still demonstrate reasonable learning and knowledge. She extended the analogy to artificial intelligence/machine learning. While a neural network is still fairly "fuzzy," it can still produce reasonable and sometimes surprising results.
  
  Of course, at the risk of overfitting, the more we train a neural network the less fuzzy it becomes.
  
  But I think it is a poor analogy, given that a JPEG is a static image and a neural network is proven to be Turing complete (computationally universal).
  
  > **Comments:**
  > **username_4**: I think some mixup on "Turing complete". The universal approximation theorem says nothing about that. I believe that recurrent neural networks can be viewed as trainable Turing machines, but not sure if there's proof of universal computation? If you have a reference for the proof you mention at the end, I would appreciate a link  
  > **username_3**: The Universal approximation theorem only addresses continuous functions, not sufficient to be Turing complete.  I have several references to papers that address neural networks as Turing complete, therefore also computational universal. I have a list of attributions to papers that prove certain classes and configurations of ANNs as Turing complete. I will share the links to the online versions I can find.  
  > **username_3**: Here are a few relevant papers addressing the topic of Turing completeness of ANNs, there are many more. "How Powerful are Decoder-Only Transformer Neural Models?"  https://arxiv.org/pdf/2305.17026.pdf "On the Turing Completeness of Modern Neural Network Architecture" https://arxiv.org/abs/1901.03429 "Attention is Turing Complete" https://jmlr.org/papers/volume22/20-302/20-302.pdf  
  
  ---
  
  