Answers.Body: <p>Well, there are some questions here...</p>
  <blockquote>
  <p>Does it (Deep Learning) try to learn a continuous distribution based
  on the training-set and its corresponding mappings, and map unseen
  examples from this learned distribution?</p>
  </blockquote>
  <p>Yes. Talking about Deep Artificial Neural Networks, they try to learn continuous distribution using continuous activation functions in each neuron. Therefore, the output is also a continuous function to represent a continuous probability distribution. The issue with the unseen examples is the need for similar examples in the training set; otherwise, the weights and bias of the network will not be tuned in the regions of space around the unseen example. Imagine a Neural Network learning a function y = x, if we only present values between 0 and 10 during training, we should expect it to only make good predictions for y for values of x ranging from 0 to 10. It doesn't mean that it won't predict for other values, but the predictions will not be so accurate or nowhere close to the expectations. That is because the network is not trying to guess what was the function used to generate y, but it is simply trying to adjust its parameters to make its internal functions generate the expected y for the given x. That is why Deep Neural Networks require a lot of data. In a unidimensional space is easier to provide examples that cover the subset of the domain we want our network to learn. When we use multidimensional space, we need a lot more examples to have a good representation of the hyperspace used as domain.</p>
  <blockquote>
  <p>Could this (map unseen examples) be said about all predictive algorithms?</p>
  </blockquote>
  <p>Yes, it should. Otherwise, the algorithm would not be able to generalize well. A good predictive algorithm is the one that can predict unseen examples using fewer training samples.</p>
  <blockquote>
  <p>Could the idea of classification problems using hyper-planes be
  extended in general to any Deep Learning problem learning a mapping?</p>
  </blockquote>
  <p>In the case of Deep Neural Networks, the result is more like, for a given input value, return the probability of it belonging to a class. For binary classification, the network will have a single output. The sigmoid function modulates this output to ranges between 0 and 1. We can interpret the output as the probability of belonging to one out of two possible classes. To know the probability for the other class, we subtract it from 1. For three or more classes, we will need three or more outputs ranging from 0 to 1, and each output is the probability of belonging to one of the classes. In this case, the outputs are also normalized by a softmax function, that guarantees that the sum of all outputs is equal to 1, as a probability distribution.</p>
  <blockquote>
  <p>Would also explain why Deep Learning needs a lot of data and why it
  works better mostly than other Learning algorithms for simple
  problems.</p>
  </blockquote>
  <p>Already partially explained... The need for a lot of data is to have a good representation of the hyperplane used as the domain.
  The Deep Neural Networks work well because of their power to represent different models. They are a very 'flexible' functions that can be bent to approximate the relation existent between the data in the training set and the expected target. Simpler algorithms, as linear models, for instance, have less representation power, they are limited to a smaller set of models. Even though many models can be linearly approximated (because the input and output almost follow a linear relation), the neural network will be able to learn the nuances of the dataset better. This can also be the curse of neural networks, because they may try to learn every detail of the training set that wasn't really relevant and true for other cases and this concept is called overtraining... but is a discussion for another topic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T10:47:10.703Z
Answers.Text: Yes, they are different indeed. I've updated the answer to try to make the difference more clear. Thank you for pointing that.
Answers.UserDisplayName: Mark C. F. Sousa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T13:36:04.683Z
Answers.Text: _Continuous distribution_ and _continuous function_ aren't exactly interchangeable expressions, but you're using them interchangeably (at least, in your first paragraphs).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-18T16:40:08.983Z
Answers.LastActivityDate: 2020-06-24T10:45:26.147Z
Answers.LastEditDate: 2020-06-24T10:45:26.147Z
Answers.UserDisplayName: Mark C. F. Sousa
Body: <p>At least at some level, maybe not end-to-end always, but deep learning always learns a function, essentially a mapping from a domain to a range. The domain and range, at least in most cases, would be multi-variate. </p>
  
  <p>So, when a model learns a mapping, considering every point in the domain-space has a mapping, does it try to learn a continuous distribution based on the training-set and its corresponding mappings, and map unseen examples from this learned distribution? Could this be said about all predictive algorithms? </p>
  
  <p>If yes, then could binary classification be compared to having a hyper-plane (as in support vector classification) in a particular kernel-space, and could the idea of classification problems using hyper-planes be extended in general to any deep learning problem learning a mapping? </p>
  
  <p>It would also explain why deep learning needs a lot of data and why it works better than other learning algorithms for simple problems.</p>
  
Tags: <neural-networks><deep-learning><probability-distribution><computational-learning-theory>
Title: In deep learning, do we learn a continuous distribution based on the training dataset?
UserDisplayName: ashenoy
ThreadText: # In deep learning, do we learn a continuous distribution based on the training dataset?
  
  (Asked by: username_0 on 2019-09-18)
  
  At least at some level, maybe not end-to-end always, but deep learning always learns a function, essentially a mapping from a domain to a range. The domain and range, at least in most cases, would be multi-variate.
  
  So, when a model learns a mapping, considering every point in the domain-space has a mapping, does it try to learn a continuous distribution based on the training-set and its corresponding mappings, and map unseen examples from this learned distribution? Could this be said about all predictive algorithms?
  
  If yes, then could binary classification be compared to having a hyper-plane (as in support vector classification) in a particular kernel-space, and could the idea of classification problems using hyper-planes be extended in general to any deep learning problem learning a mapping?
  
  It would also explain why deep learning needs a lot of data and why it works better than other learning algorithms for simple problems.
  
  **Tags:** neural-networks, deep-learning, probability-distribution, computational-learning-theory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-18. Score: 3 | Accepted answer
  
  Well, there are some questions here...
  
  > Does it (Deep Learning) try to learn a continuous distribution based
  > on the training-set and its corresponding mappings, and map unseen
  > examples from this learned distribution?
  
  Yes. Talking about Deep Artificial Neural Networks, they try to learn continuous distribution using continuous activation functions in each neuron. Therefore, the output is also a continuous function to represent a continuous probability distribution. The issue with the unseen examples is the need for similar examples in the training set; otherwise, the weights and bias of the network will not be tuned in the regions of space around the unseen example. Imagine a Neural Network learning a function y = x, if we only present values between 0 and 10 during training, we should expect it to only make good predictions for y for values of x ranging from 0 to 10. It doesn't mean that it won't predict for other values, but the predictions will not be so accurate or nowhere close to the expectations. That is because the network is not trying to guess what was the function used to generate y, but it is simply trying to adjust its parameters to make its internal functions generate the expected y for the given x. That is why Deep Neural Networks require a lot of data. In a unidimensional space is easier to provide examples that cover the subset of the domain we want our network to learn. When we use multidimensional space, we need a lot more examples to have a good representation of the hyperspace used as domain.
  
  > Could this (map unseen examples) be said about all predictive algorithms?
  
  Yes, it should. Otherwise, the algorithm would not be able to generalize well. A good predictive algorithm is the one that can predict unseen examples using fewer training samples.
  
  > Could the idea of classification problems using hyper-planes be
  > extended in general to any Deep Learning problem learning a mapping?
  
  In the case of Deep Neural Networks, the result is more like, for a given input value, return the probability of it belonging to a class. For binary classification, the network will have a single output. The sigmoid function modulates this output to ranges between 0 and 1. We can interpret the output as the probability of belonging to one out of two possible classes. To know the probability for the other class, we subtract it from 1. For three or more classes, we will need three or more outputs ranging from 0 to 1, and each output is the probability of belonging to one of the classes. In this case, the outputs are also normalized by a softmax function, that guarantees that the sum of all outputs is equal to 1, as a probability distribution.
  
  > Would also explain why Deep Learning needs a lot of data and why it
  > works better mostly than other Learning algorithms for simple
  > problems.
  
  Already partially explained... The need for a lot of data is to have a good representation of the hyperplane used as the domain.
  The Deep Neural Networks work well because of their power to represent different models. They are a very 'flexible' functions that can be bent to approximate the relation existent between the data in the training set and the expected target. Simpler algorithms, as linear models, for instance, have less representation power, they are limited to a smaller set of models. Even though many models can be linearly approximated (because the input and output almost follow a linear relation), the neural network will be able to learn the nuances of the dataset better. This can also be the curse of neural networks, because they may try to learn every detail of the training set that wasn't really relevant and true for other cases and this concept is called overtraining... but is a discussion for another topic.
  
  > **Comments:**
  > **username_2**: _Continuous distribution_ and _continuous function_ aren't exactly interchangeable expressions, but you're using them interchangeably (at least, in your first paragraphs).  
  > **username_1**: Yes, they are different indeed. I've updated the answer to try to make the difference more clear. Thank you for pointing that.  
  
  ---
  
  