Body: <p>I've tried to come up with a simple connection between statistics and deep learning. One question is:</p>
  <ul>
  <li>In very simplified terms, <em>why do neural networks model probability distributions?</em></li>
  </ul>
  <p>What I expect is an authoritative answer as to how this chain of thinking answers or not the above question, and whether it's directionally correct or not.</p>
  <hr />
  <p>This is a minor answer to check with others whether this is directionally correct.</p>
  <p>Had we just a line like so:</p>
  <p><a href="https://i.sstatic.net/0bEbRZtC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0bEbRZtC.png" alt="enter image description here" /></a></p>
  <p>Then that's fully determined and there isn't any uncertainty on <code>x</code> or <code>y</code>.</p>
  <p>However, in this next case it's different:</p>
  <p><a href="https://i.sstatic.net/KPbeKDBG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KPbeKDBG.png" alt="enter image description here" /></a></p>
  <p>The <code>x</code> values are distributed with some probability along <code>x</code>, and the correspondent is valid for <code>y</code>.</p>
  <p>In some cases the probability on each axis may approach a Gaussian distribution, which I would draw as:</p>
  <p><a href="https://i.sstatic.net/9iLkX9KN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9iLkX9KN.png" alt="enter image description here" /></a></p>
  <p>Obviously this does not need be the case, it's just an example.</p>
  <p>If I understand correctly, both <code>x</code> and <code>y</code> can be considered random variables <code>X</code> and <code>Y</code>, and in that case, it's easy to see why neural networks learn probability distributions, what they actually learn is the function approximating the mapping (that reduces the error.)</p>
  <p>In other words, neural networks model probability distributions.</p>
  <p>So it should be clear there why one talks about statistics in deep learning, it's mostly about the spread in <code>X</code>, and <code>Y</code>. <code>f: X ~=&gt; Y</code> Where <code>f(X)</code> is the complicated function / neural network. (In the learning process that's actually the Loss.)</p>
  <p>As described in a similar way in <a href="https://en.wikipedia.org/wiki/Loss_function" rel="nofollow noreferrer">Wikipedia for the Loss function</a>:</p>
  <p><a href="https://i.sstatic.net/fTwAQh6t.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fTwAQh6t.png" alt="enter image description here" /></a></p>
  <p>Is this very simplified understanding correct at least directionally ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-18T00:16:52.107Z
Comments.Text: [A related, possibly duplicate, question is on Cross Validated, to which I posted an answer I like.](https://stats.stackexchange.com/q/647266/247274)
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-18T11:51:03.577Z
Comments.Text: what im looking for isn't really in that post, but it's an interesting one. what im trying to represent is much closer to https://en.wikipedia.org/wiki/Empirical_risk_minimization
Comments.UserDisplayName: Minsky
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-18T18:49:25.427Z
Comments.Text: What about that answer of mine fails to answer the question you’ve posted here (and yesterday)? I give multiple examples of full conditional distributions mattering (medical diagnosis, investing) and discuss how considering conditional distributions can lead to estimation strategies.
Comments.UserDisplayName: Dave
Tags: <deep-learning><random-variable>
Title: Why neural networks model statistical patterns
UserDisplayName: Minsky
ThreadText: # Why neural networks model statistical patterns
  
  (Asked by: username_0 on 2024-08-17)
  
  I've tried to come up with a simple connection between statistics and deep learning. One question is:
  
  * In very simplified terms, *why do neural networks model probability distributions?*
  
  What I expect is an authoritative answer as to how this chain of thinking answers or not the above question, and whether it's directionally correct or not.
  
  ---
  
  This is a minor answer to check with others whether this is directionally correct.
  
  Had we just a line like so:
  
  [![enter image description here](https://i.sstatic.net/0bEbRZtC.png)](https://i.sstatic.net/0bEbRZtC.png)
  
  Then that's fully determined and there isn't any uncertainty on `x` or `y`.
  
  However, in this next case it's different:
  
  [![enter image description here](https://i.sstatic.net/KPbeKDBG.png)](https://i.sstatic.net/KPbeKDBG.png)
  
  The `x` values are distributed with some probability along `x`, and the correspondent is valid for `y`.
  
  In some cases the probability on each axis may approach a Gaussian distribution, which I would draw as:
  
  [![enter image description here](https://i.sstatic.net/9iLkX9KN.png)](https://i.sstatic.net/9iLkX9KN.png)
  
  Obviously this does not need be the case, it's just an example.
  
  If I understand correctly, both `x` and `y` can be considered random variables `X` and `Y`, and in that case, it's easy to see why neural networks learn probability distributions, what they actually learn is the function approximating the mapping (that reduces the error.)
  
  In other words, neural networks model probability distributions.
  
  So it should be clear there why one talks about statistics in deep learning, it's mostly about the spread in `X`, and `Y`. `f: X ~=> Y` Where `f(X)` is the complicated function / neural network. (In the learning process that's actually the Loss.)
  
  As described in a similar way in [Wikipedia for the Loss function](https://en.wikipedia.org/wiki/Loss_function):
  
  [![enter image description here](https://i.sstatic.net/fTwAQh6t.png)](https://i.sstatic.net/fTwAQh6t.png)
  
  Is this very simplified understanding correct at least directionally ?
  
  **Tags:** deep-learning, random-variable
  
  > **Comments:**
  > **username_1**: [A related, possibly duplicate, question is on Cross Validated, to which I posted an answer I like.](https://stats.stackexchange.com/q/647266/247274)  
  > **username_0**: what im looking for isn't really in that post, but it's an interesting one. what im trying to represent is much closer to https://en.wikipedia.org/wiki/Empirical_risk_minimization  
  > **username_1**: What about that answer of mine fails to answer the question you’ve posted here (and yesterday)? I give multiple examples of full conditional distributions mattering (medical diagnosis, investing) and discuss how considering conditional distributions can lead to estimation strategies.  
  
  ---
  
  ## 0 answers
  
  