Answers.Body: <p>According to Wikipedia:</p>
  
  <blockquote>
    <p>A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generating process.</p>
  </blockquote>
  
  <p>Answer to your question:</p>
  
  <p>To build any neural network model we assume the train, test and validation data are coming from a probability distribution. So, if you produce a neural network model based on statistical data then the network is a statistical model. </p>
  
  <p>Moreover, neural networks' cost function is generally a parametric model and parametric modes are statistical models.  </p>
  
  <p>Please look at Goodfellow's Deep Learning book chapter <a href="https://www.deeplearningbook.org/contents/mlp.html" rel="noreferrer">Deep Feedforward Networks</a> page 174 and 175.</p>
  
  <p>From Goodfellow's book </p>
  
  <blockquote>
    <p>Fortunately, the cost functions for neural networks are more or less the same as those for other parametric models, such as linear models. In most cases, our <a href="https://en.wikipedia.org/wiki/Parametric_model" rel="noreferrer">parametric model</a> deﬁnes a distribution <span class="math-container">$p(y \mid x; \theta)$</span> and we simply use the principle of maximum likelihood. </p>
  </blockquote>
  
  <p>In conclusion, ANNs (e. g. MLP, CNN, etc.) are statistical models</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-12T03:06:52.870Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105466/discussion-on-answer-by-ta-req-are-neural-networks-statistical-models).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-12T01:26:33.493Z
Answers.LastActivityDate: 2020-03-12T03:05:38.527Z
Answers.LastEditDate: 2020-03-12T03:05:38.527Z
Answers.UserDisplayName: Ta_Req
Answers.Body: <p>A dataset can be thought of as a set of ordered pairs <span class="math-container">$\subset R^f \times R^l$</span>,  where <span class="math-container">$f$</span> is the feature dimensions and <span class="math-container">$l$</span> the label dimensions.</p>
  <p>Ordered pairs give rise to a statistical model (i.e. a function from <span class="math-container">$R^f$</span> to a probability distribution over <span class="math-container">$R^l$</span>)</p>
  <p>Then this statistical model is turned into function <span class="math-container">$R^f \mapsto R^l$</span> by taking the argmax.</p>
  <p>A neural network (trained on a dataset) is a compression of this statistical model composed with argmax. (think of how much smaller the GPT3 weights are than the dataset).</p>
  <p>So they are approximations to statistical models composed with argmax.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-26T05:17:54.233Z
Answers.LastActivityDate: 2022-10-26T05:17:54.233Z
Answers.UserDisplayName: Tom Huntington
Answers.Body: <h2>What is a statistical model?</h2>
  <p>According to Anthony C. Davison (in the book <a href="https://fac.ksu.edu.sa/sites/default/files/statistical_models.pdf" rel="nofollow noreferrer">Statistical Models</a>), a statistical model is a probability distribution constructed to enable inferences to be drawn or decisions made from data. The probability distribution represents the variability of the data.</p>
  <h2>Are neural networks statistical models?</h2>
  <p>Do neural networks construct or represent a probability distribution that enables inferences to be drawn or decisions made from data?</p>
  <h3>MLP for binary classification</h3>
  <p>For example, a multi-layer perceptron (MLP) trained to solve a binary classification task can be thought of as model of the probability distribution <span class="math-container">$\mathbb{P}(y \mid x; \theta)$</span>. In fact, there are many examples of MLPs with a softmax or sigmoid function as the activation function of the output layer in order to produce a probability or a probability vector. However, it's important to note that, although many neural networks produce a probability or a probability vector, a probability distribution is not the same thing. A probability alone does not describe a full probability distribution and different distributions are defined by different parameters (e.g. a Bernoulli is defined by <span class="math-container">$p$</span>, while a Gaussian by <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma$</span>). However, for example, if you make your neural network produce a probability, i.e. model <span class="math-container">$\mathbb{P}(y = 1 \mid x; \theta)$</span>, at least in the case of binary classification, you could obviously derive the probability of the other label as follows: <span class="math-container">$\mathbb{P}(y = 0 \mid x; \theta) = 1 - \mathbb{P}(y = 1 \mid x; \theta)$</span>. In any case, in this example, you only need the parameter <span class="math-container">$p = \mathbb{P}(y = 1 \mid x; \theta)$</span> to define the associated <a href="https://en.wikipedia.org/wiki/Bernoulli_distribution" rel="nofollow noreferrer">Bernoulli distribution</a>.</p>
  <p>So, these neural networks (for binary classification) that model and learn some probability distribution given the data in order to make inferences or predictions <em>could</em> be considered statistical models. However, note that, once the weights of the neural network are fixed, given the same input, they always produce the same output.</p>
  <h3>Generative models</h3>
  <p>Variational auto-encoders (VAEs) construct a probability distribution (e.g. a Gaussian or <span class="math-container">$\mathbb{P}(x)$</span> that represents the probability distribution over images, if you want to generate images), so VAEs <em>can</em> be considered statistical models.</p>
  <h3>Bayesian neural networks</h3>
  <p>There are also <a href="http://proceedings.mlr.press/v37/blundell15.html" rel="nofollow noreferrer">Bayesian neural networks</a>, which are neural networks that maintain a probability distribution (usually, a Gaussian) for each unit (or neuron) of the neural network, rather than only a point estimate. BNNs <em>can</em> thus also be considered statistical models.</p>
  <h3>Perceptron</h3>
  <p>The perceptron may be considered a &quot;statistical model&quot;, in the sense that it learns from data, but it doesn't produce any probability vector or distribution, i.e. it is not a probabilistic model/classifier.</p>
  <h2>Conclusion</h2>
  <p>So, whether or not a neural network is a statistical model depends on your definition of a statistical model and which machine learning models you would consider neural networks. If you are interested in more formal definitions of a statistical model, take a look at <a href="http://www.stat.uchicago.edu/%7Epmcc/pubs/AOS023.pdf" rel="nofollow noreferrer">this paper</a>.</p>
  <h2>Parametric vs non-parametric</h2>
  <p>Statistical models are often also divided into parametric and non-parametric models. Neural networks are often classified as non-parametric because they make fewer assumptions than e.g. linear regression models (which are parametric) and are typically more generally applicable, but I will not dwell on this topic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-02T12:18:45.850Z
Answers.Text: This is a great and detailed answer! I have a question - one can have an assumption on the probabilistic nature of the data, on which the model is trained, and search for a single optimal solution.  On the other hand, a probability distribution can be imposed on the model params (like for mentioned Bayesian NN) - and the result would be like mean and the std of the optimal params. Is there any specific name for this kind of problem?
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-02T14:55:24.633Z
Answers.Text: @spiridon_the_sun_rotator Are you asking whether there's a sub-field of ML that studies Bayesian neural networks? If that's the question, the name is usually "Bayesian deep learning", or "Bayesian ML" or even "Probabilistic ML", but note that these terms may not just refer to techniques like BNNs, but other things too, like Bayesian optimization or graphical models, as far as I remember.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-16T14:57:00.340Z
Answers.LastActivityDate: 2022-06-04T14:15:06.297Z
Answers.LastEditDate: 2022-06-04T14:15:06.297Z
Answers.UserDisplayName: nbro
Body: <p>By reading the abstract of <a href="https://people.orie.cornell.edu/davidr/or474/nn_sas.pdf" rel="nofollow noreferrer">Neural Networks and Statistical Models</a> paper it would seem that ANNs are statistical models.</p>
  
  <p>In contrast <a href="https://towardsdatascience.com/no-machine-learning-is-not-just-glorified-statistics-26d3952234e3" rel="nofollow noreferrer">Machine Learning is not just glorified Statistics</a>.</p>
  
  <p>I am looking for a more concise/summarized answer with focus on ANNs.</p>
  
Tags: <neural-networks><machine-learning><deep-learning><terminology><statistical-ai>
Title: Are neural networks statistical models?
UserDisplayName: Leo Gallucci
ThreadText: # Are neural networks statistical models?
  
  (Asked by: username_0 on 2019-01-29)
  
  By reading the abstract of [Neural Networks and Statistical Models](https://people.orie.cornell.edu/davidr/or474/nn_sas.pdf) paper it would seem that ANNs are statistical models.
  
  In contrast [Machine Learning is not just glorified Statistics](https://towardsdatascience.com/no-machine-learning-is-not-just-glorified-statistics-26d3952234e3).
  
  I am looking for a more concise/summarized answer with focus on ANNs.
  
  **Tags:** neural-networks, machine-learning, deep-learning, terminology, statistical-ai
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2019-05-16. Score: 7 | Accepted answer
  
  ## What is a statistical model?
  
  According to Anthony C. Davison (in the book [Statistical Models](https://fac.ksu.edu.sa/sites/default/files/statistical_models.pdf)), a statistical model is a probability distribution constructed to enable inferences to be drawn or decisions made from data. The probability distribution represents the variability of the data.
  
  ## Are neural networks statistical models?
  
  Do neural networks construct or represent a probability distribution that enables inferences to be drawn or decisions made from data?
  
  ### MLP for binary classification
  
  For example, a multi-layer perceptron (MLP) trained to solve a binary classification task can be thought of as model of the probability distribution $\mathbb{P}(y \mid x; \theta)$. In fact, there are many examples of MLPs with a softmax or sigmoid function as the activation function of the output layer in order to produce a probability or a probability vector. However, it's important to note that, although many neural networks produce a probability or a probability vector, a probability distribution is not the same thing. A probability alone does not describe a full probability distribution and different distributions are defined by different parameters (e.g. a Bernoulli is defined by $p$, while a Gaussian by $\mu$ and $\sigma$). However, for example, if you make your neural network produce a probability, i.e. model $\mathbb{P}(y = 1 \mid x; \theta)$, at least in the case of binary classification, you could obviously derive the probability of the other label as follows: $\mathbb{P}(y = 0 \mid x; \theta) = 1 - \mathbb{P}(y = 1 \mid x; \theta)$. In any case, in this example, you only need the parameter $p = \mathbb{P}(y = 1 \mid x; \theta)$ to define the associated [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution).
  
  So, these neural networks (for binary classification) that model and learn some probability distribution given the data in order to make inferences or predictions *could* be considered statistical models. However, note that, once the weights of the neural network are fixed, given the same input, they always produce the same output.
  
  ### Generative models
  
  Variational auto-encoders (VAEs) construct a probability distribution (e.g. a Gaussian or $\mathbb{P}(x)$ that represents the probability distribution over images, if you want to generate images), so VAEs *can* be considered statistical models.
  
  ### Bayesian neural networks
  
  There are also [Bayesian neural networks](http://proceedings.mlr.press/v37/blundell15.html), which are neural networks that maintain a probability distribution (usually, a Gaussian) for each unit (or neuron) of the neural network, rather than only a point estimate. BNNs *can* thus also be considered statistical models.
  
  ### Perceptron
  
  The perceptron may be considered a "statistical model", in the sense that it learns from data, but it doesn't produce any probability vector or distribution, i.e. it is not a probabilistic model/classifier.
  
  ## Conclusion
  
  So, whether or not a neural network is a statistical model depends on your definition of a statistical model and which machine learning models you would consider neural networks. If you are interested in more formal definitions of a statistical model, take a look at [this paper](http://www.stat.uchicago.edu/%7Epmcc/pubs/AOS023.pdf).
  
  ## Parametric vs non-parametric
  
  Statistical models are often also divided into parametric and non-parametric models. Neural networks are often classified as non-parametric because they make fewer assumptions than e.g. linear regression models (which are parametric) and are typically more generally applicable, but I will not dwell on this topic.
  
  > **Comments:**
  > **username_4**: This is a great and detailed answer! I have a question - one can have an assumption on the probabilistic nature of the data, on which the model is trained, and search for a single optimal solution.  On the other hand, a probability distribution can be imposed on the model params (like for mentioned Bayesian NN) - and the result would be like mean and the std of the optimal params. Is there any specific name for this kind of problem?  
  > **username_2**: @spiridon_the_sun_rotator Are you asking whether there's a sub-field of ML that studies Bayesian neural networks? If that's the question, the name is usually "Bayesian deep learning", or "Bayesian ML" or even "Probabilistic ML", but note that these terms may not just refer to techniques like BNNs, but other things too, like Bayesian optimization or graphical models, as far as I remember.  
  
  ---
  
  ### Answer by username_1 on 2020-03-12. Score: 5
  
  According to Wikipedia:
  
  > A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generating process.
  
  Answer to your question:
  
  To build any neural network model we assume the train, test and validation data are coming from a probability distribution. So, if you produce a neural network model based on statistical data then the network is a statistical model.
  
  Moreover, neural networks' cost function is generally a parametric model and parametric modes are statistical models.
  
  Please look at Goodfellow's Deep Learning book chapter [Deep Feedforward Networks](https://www.deeplearningbook.org/contents/mlp.html) page 174 and 175.
  
  From Goodfellow's book
  
  > Fortunately, the cost functions for neural networks are more or less the same as those for other parametric models, such as linear models. In most cases, our [parametric model](https://en.wikipedia.org/wiki/Parametric_model) deﬁnes a distribution $p(y \mid x; \theta)$ and we simply use the principle of maximum likelihood.
  
  In conclusion, ANNs (e. g. MLP, CNN, etc.) are statistical models
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105466/discussion-on-answer-by-ta-req-are-neural-networks-statistical-models).  
  
  ---
  
  ### Answer by username_3 on 2022-10-26. Score: 2
  
  A dataset can be thought of as a set of ordered pairs $\subset R^f \times R^l$, where $f$ is the feature dimensions and $l$ the label dimensions.
  
  Ordered pairs give rise to a statistical model (i.e. a function from $R^f$ to a probability distribution over $R^l$)
  
  Then this statistical model is turned into function $R^f \mapsto R^l$ by taking the argmax.
  
  A neural network (trained on a dataset) is a compression of this statistical model composed with argmax. (think of how much smaller the GPT3 weights are than the dataset).
  
  So they are approximations to statistical models composed with argmax.
  
  
  ---
  
  