Answers.Body: <p><strong>Intuitively</strong>, this is similar to the case when you are making predictions but you don't have all the necessary information to make the most accurate prediction or maybe there isn't a single accurate prediction, so you have a set of possible predictions (rather than a single prediction). </p>
  
  <p>For example, if you hadn't seen <a href="https://www.uefa.com/uefachampionsleague/match/2027125--atletico-vs-liverpool/?iv=true" rel="nofollow noreferrer">the last Liverpool game</a> (in the Champions League) against Atlético Madrid, you would have probably said that Liverpool was the most likely team to win the CL this year (2020) too. However, after having seen their last game, you noticed that they are not unbeatable and they are not perfect, so, although they have shown you (during this and the previous season) that they are a very good team, they may also no be the best until the end of the season. So, at this point, you may have a set of two possible hypotheses: Liverpool will win the CL or Liverpool will not win the CL.</p>
  
  <p>In general, if you had a dataset that is representative of your whole population, then the dataset alone should be sufficient to make accurate predictions (i.e. it contains all the information sufficient to make accurate predictions). If that's not the case (which is often true), then you will have to account for all possible values of the missing data or you will have to make assumptions (or introduce an <a href="https://en.wikipedia.org/wiki/Inductive_bias" rel="nofollow noreferrer">inductive bias</a>).</p>
  
  <p>The authors also mention the concept of <a href="https://en.wikipedia.org/wiki/Marginal_distribution" rel="nofollow noreferrer">marginalization</a>, which is used in probability theory to calculate marginal probabilities, e.g. <span class="math-container">$p(X=x)$</span> (or for short <span class="math-container">$p(x)$</span>), when there's another random variable <span class="math-container">$Y$</span>, by accounting for all possible values of <span class="math-container">$Y$</span>. In other words, you're interested <strong>only</strong> in <span class="math-container">$p(x)$</span> and you may have the joint probability distribution <span class="math-container">$p(x, y)$</span>, then marginalization allows you to compute <span class="math-container">$p(x)$</span> using e.g. <span class="math-container">$p(x, y)$</span> and all possible values that the random variable <span class="math-container">$Y$</span> can take.</p>
  
  <p>In any case, I think their description is a little bit vague and using the concept of marginalization to convey the idea behind the "multiple hypotheses" isn't the most appropriate approach, IMHO. If you are interested in these concepts in the context of neural networks, I suggest you read something about Bayesian machine learning or Bayesian neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-03T22:31:39.873Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/119296/discussion-on-answer-by-nbro-why-does-the-machine-learning-algorithm-need-to-lea).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T12:41:35.703Z
Answers.LastActivityDate: 2020-02-20T13:25:14.010Z
Answers.LastEditDate: 2020-02-20T13:25:14.010Z
Answers.UserDisplayName: nbro
Answers.Body: <p>@The Pointer the <span class="math-container">$2^n$</span> came from the question: How many function do we need to have if each of the <span class="math-container">$n$</span> inputs can be missing?
  example: <span class="math-container">$f_1(\text{missing}, x_2, x_3, \dots, x_n)$</span> for <span class="math-container">$x_1$</span> missing
  <span class="math-container">$f_2(x_1, x_2, \text{missing}, x_4, \text{missing}, \dots, x_n)$</span> for <span class="math-container">$x_3$</span> and <span class="math-container">$x_5$</span> missing.</p>
  <p>So this problem is a combinatorial one and the event for each <span class="math-container">$x_i$</span> is Missing or Not.
  Each function corresponds 1-by-1 with a possible set <span class="math-container">$(x_1, \text{missing}, x_2, \dots, x_n)$</span>. So how many sets can you form <span class="math-container">$2^n$</span>. Why?</p>
  <p>This formula comes back to like tree developing. First variable <span class="math-container">$x_1$</span> missing or not (2 possbile events). Now after that, for EACH of these events 2 possbile events for <span class="math-container">$x_2$</span> so in total 2 \times 2 (2 for <span class="math-container">$x_1$</span> , 2 for <span class="math-container">$x_2$</span> for each <span class="math-container">$x_1$</span>'s event) and etc., <span class="math-container">$2 \times 2 \times 2 \times \dots$</span> <span class="math-container">$n$</span> times = <span class="math-container">$2^n$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-04T10:31:20.153Z
Answers.Text: Thanks for the clarification. Your post is helpful, but, since it doesn't address the main question, I think it would have been more suitable as a comment.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-03T22:06:41.433Z
Answers.LastActivityDate: 2021-02-04T10:57:24.727Z
Answers.LastEditDate: 2021-02-04T10:57:24.727Z
Answers.UserDisplayName: Fabrice Simo Defo
Body: <p>I am currently studying the textbook <em>Deep Learning</em> by Goodfellow, Bengio, and Courville. Chapter <strong>5.1 Learning Algorithms</strong> says the following:</p>
  
  <blockquote>
    <p><strong>Classification with missing inputs</strong>: Classification becomes more challenging if the computer program is not guaranteed that every measurement in its input vector will always be provided. To solve the classification task, the learning algorithm only has to define a single function mapping from a vector input to a categorical output. When some of the inputs may be missing, rather than providing a single classification function, the learning algorithm must learn a set of functions. Each function corresponds to classifying <span class="math-container">$\mathbf{x}$</span> with a different subset of its inputs missing. This kind of situation arises frequently in medical diagnosis, because many kinds of medical tests are expensive or invasive. One way to efficiently define such a large set of functions is to learn a probability distribution over all the relevant variables, then solve the classification task by marginalizing out the missing variables. With <span class="math-container">$n$</span> input variables, we can now obtain all <span class="math-container">$2^n$</span> different classification functions needed for each possible set of missing inputs, but the computer program needs to learn only a single function describing the joint probability distribution. See Goodfellow et al. (2013b) for an example of a deep probabilistic model applied to such a task in this way. Many of the other tasks described in this section can also be generalized to work with missing inputs; classification with missing inputs is just one example of what machine learning can do. </p>
  </blockquote>
  
  <p>I was wondering if people would please help me better understand this explanation. Why is it that, when some of the inputs are missing, rather than providing a single classification function, the learning algorithm must learn a set of functions? And what is meant by "each function corresponds to classifying <span class="math-container">$\mathbf{x}$</span> with a different subset of its inputs missing."?</p>
  
  <p>I would greatly appreciate it if people would please take the time to clarify this.</p>
  
Tags: <machine-learning><classification><learning-algorithms><probability-distribution>
Title: Why does the machine learning algorithm need to learn a set of functions in the case of missing data?
UserDisplayName: The Pointer
ThreadText: # Why does the machine learning algorithm need to learn a set of functions in the case of missing data?
  
  (Asked by: username_0 on 2020-02-20)
  
  I am currently studying the textbook *Deep Learning* by Goodfellow, Bengio, and Courville. Chapter **5.1 Learning Algorithms** says the following:
  
  > **Classification with missing inputs**: Classification becomes more challenging if the computer program is not guaranteed that every measurement in its input vector will always be provided. To solve the classification task, the learning algorithm only has to define a single function mapping from a vector input to a categorical output. When some of the inputs may be missing, rather than providing a single classification function, the learning algorithm must learn a set of functions. Each function corresponds to classifying $\mathbf{x}$ with a different subset of its inputs missing. This kind of situation arises frequently in medical diagnosis, because many kinds of medical tests are expensive or invasive. One way to efficiently define such a large set of functions is to learn a probability distribution over all the relevant variables, then solve the classification task by marginalizing out the missing variables. With $n$ input variables, we can now obtain all $2^n$ different classification functions needed for each possible set of missing inputs, but the computer program needs to learn only a single function describing the joint probability distribution. See Goodfellow et al. (2013b) for an example of a deep probabilistic model applied to such a task in this way. Many of the other tasks described in this section can also be generalized to work with missing inputs; classification with missing inputs is just one example of what machine learning can do.
  
  I was wondering if people would please help me better understand this explanation. Why is it that, when some of the inputs are missing, rather than providing a single classification function, the learning algorithm must learn a set of functions? And what is meant by "each function corresponds to classifying $\mathbf{x}$ with a different subset of its inputs missing."?
  
  I would greatly appreciate it if people would please take the time to clarify this.
  
  **Tags:** machine-learning, classification, learning-algorithms, probability-distribution
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-20. Score: 2 | Accepted answer
  
  **Intuitively**, this is similar to the case when you are making predictions but you don't have all the necessary information to make the most accurate prediction or maybe there isn't a single accurate prediction, so you have a set of possible predictions (rather than a single prediction).
  
  For example, if you hadn't seen [the last Liverpool game](https://www.uefa.com/uefachampionsleague/match/2027125--atletico-vs-liverpool/?iv=true) (in the Champions League) against Atlético Madrid, you would have probably said that Liverpool was the most likely team to win the CL this year (2020) too. However, after having seen their last game, you noticed that they are not unbeatable and they are not perfect, so, although they have shown you (during this and the previous season) that they are a very good team, they may also no be the best until the end of the season. So, at this point, you may have a set of two possible hypotheses: Liverpool will win the CL or Liverpool will not win the CL.
  
  In general, if you had a dataset that is representative of your whole population, then the dataset alone should be sufficient to make accurate predictions (i.e. it contains all the information sufficient to make accurate predictions). If that's not the case (which is often true), then you will have to account for all possible values of the missing data or you will have to make assumptions (or introduce an [inductive bias](https://en.wikipedia.org/wiki/Inductive_bias)).
  
  The authors also mention the concept of [marginalization](https://en.wikipedia.org/wiki/Marginal_distribution), which is used in probability theory to calculate marginal probabilities, e.g. $p(X=x)$ (or for short $p(x)$), when there's another random variable $Y$, by accounting for all possible values of $Y$. In other words, you're interested **only** in $p(x)$ and you may have the joint probability distribution $p(x, y)$, then marginalization allows you to compute $p(x)$ using e.g. $p(x, y)$ and all possible values that the random variable $Y$ can take.
  
  In any case, I think their description is a little bit vague and using the concept of marginalization to convey the idea behind the "multiple hypotheses" isn't the most appropriate approach, IMHO. If you are interested in these concepts in the context of neural networks, I suggest you read something about Bayesian machine learning or Bayesian neural networks.
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/119296/discussion-on-answer-by-nbro-why-does-the-machine-learning-algorithm-need-to-lea).  
  
  ---
  
  ### Answer by username_2 on 2021-02-03. Score: 2
  
  @The Pointer the $2^n$ came from the question: How many function do we need to have if each of the $n$ inputs can be missing?
  example: $f\_1(\text{missing}, x\_2, x\_3, \dots, x\_n)$ for $x\_1$ missing
  $f\_2(x\_1, x\_2, \text{missing}, x\_4, \text{missing}, \dots, x\_n)$ for $x\_3$ and $x\_5$ missing.
  
  So this problem is a combinatorial one and the event for each $x\_i$ is Missing or Not.
  Each function corresponds 1-by-1 with a possible set $(x\_1, \text{missing}, x\_2, \dots, x\_n)$. So how many sets can you form $2^n$. Why?
  
  This formula comes back to like tree developing. First variable $x\_1$ missing or not (2 possbile events). Now after that, for EACH of these events 2 possbile events for $x\_2$ so in total 2 \times 2 (2 for $x\_1$ , 2 for $x\_2$ for each $x\_1$'s event) and etc., $2 \times 2 \times 2 \times \dots$ $n$ times = $2^n$.
  
  > **Comments:**
  > **username_0**: Thanks for the clarification. Your post is helpful, but, since it doesn't address the main question, I think it would have been more suitable as a comment.  
  
  ---
  
  