Answers.Body: <p>A hypothesis is a statement that suggests an as yet unproven explanation of a relationship between two or more phenomena that you intend to test. An agronomist thinks that more nitrogen on canola will always increase the crop output <span class="math-container">$$Harvest = f(N)$$</span>, or a meteorologist thinks he can show that the path of a hurricane over the ocean can be determined by knowledge of the sea temperature and the wind speed at an altitude of 1000 feet one minute before. <span class="math-container">$$D(t,0) = f(T(t-1,1000),S(t-1,1000)$$</span>
  Both hypotheses are pegs on which later steps are based; testing follows with a conclusion whether the hypothesis can be rejected or not.</p>
  
  <p>Changing a hypothesis can be simply adding or subtracting arguments to the function or changing the nature of the relationship such as the acceleration of the wind as opposed to its velocity.</p>
  
  <p>A "learning" algorithm describes how the parameters of a numeric model are changed in accordance with the delta rule, that is what the learning rate is and whether momentum is to be applied.</p>
  
  <p>Random Forest and Decision Tree are "classification" algorithms. They are clearly stepwise processes that proceed towards the goal of a model, but they start by specifying the shape that the model will take and place boundaries on what values the parameters may take.</p>
  
  <p>Both learning and classification algorithms specify a priori what shape the model will take and by doing so limit its relevance to particular problems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-24T19:22:54.497Z
Answers.LastActivityDate: 2019-11-24T19:22:54.497Z
Answers.UserDisplayName: Colin Beckingham
Answers.Body: <p>In <a href="http://eliassi.org/COLTSurveyArticle.pdf" rel="nofollow noreferrer">computational learning theory</a>, a learning algorithm (or learner) <span class="math-container">$A$</span> is an algorithm that chooses a hypothesis (which is a function) <span class="math-container">$h: \mathcal{X} \rightarrow \mathcal{Y}$</span>, where <span class="math-container">$\mathcal{X}$</span> is the input space and <span class="math-container">$\mathcal{Y}$</span> is the target space, from the hypothesis space <span class="math-container">$H$</span>.</p>
  <p>For example, consider the task of image classification (e.g. <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST</a>). You can train, with gradient descent, a neural network to classify the images. In this case, gradient descent is the learner <span class="math-container">$A$</span>, the space of all possible neural networks that gradient descent considers is the hypothesis space <span class="math-container">$H$</span> (so each combination of parameters of the neural network represents a specific hypothesis), <span class="math-container">$\mathcal{X}$</span> is the space of images that you want to classify, <span class="math-container">$\mathcal{Y}$</span> is the space of all possible classes and the final trained neural network is the hypothesis <span class="math-container">$h$</span> chosen by the learner <span class="math-container">$A$</span>.</p>
  <blockquote>
  <p>For example, would the decision tree and random forest be considered two different learning algorithms?</p>
  </blockquote>
  <p>The decision tree and random forest are not learning algorithms. A <strong>specific</strong> decision tree or random forest is a <strong>hypothesis</strong> (i.e. function of the form as defined above).</p>
  <p>In the context of decision trees, the <a href="https://en.wikipedia.org/wiki/ID3_algorithm" rel="nofollow noreferrer">ID3 algorithm</a> (a <a href="https://en.wikipedia.org/wiki/Decision_tree_learning" rel="nofollow noreferrer">decision tree algorithm</a> that can be used to construct the decision tree, i.e. the hypothesis), is an example of a <strong>learning algorithm</strong> (aka <em>learner</em>).</p>
  <p>The space of all trees that the learner considers is the <strong>hypothesis space/class</strong>.</p>
  <blockquote>
  <p>Would a shallow neural network (that ends up learning a linear function) and a linear regression model, both of which use gradient descent to learn parameters, be considered different learning algorithms?</p>
  </blockquote>
  <p>The same can be said here. A specific neural network or linear regression model (i.e. a line) corresponds to a specific hypothesis. The set of all neural networks (or lines, in the case of linear regression) that you consider corresponds to the hypothesis class.</p>
  <blockquote>
  <p>Anyway, from what I understand, one way to vary the hypothesis <span class="math-container">$f$</span> would be to change the parameter values, maybe even the hyper-parameter values of, say, a decision tree.</p>
  </blockquote>
  <p>If you consider a neural network (or decision tree) model, with <span class="math-container">$N$</span> parameters <span class="math-container">$\mathbf{\theta} = [\theta_i, \dots \theta_N]$</span>, then a specific combination of these parameters corresponds to a specific hypothesis. If you change the values of these parameters, you also automatically change the hypothesis. If you change the hyperparameters (such as the number of neurons in a specific layer), however, you will be changing the hypothesis class, so the set of hypotheses that you consider.</p>
  <blockquote>
  <p>Are there other ways of varying <span class="math-container">$f$</span>?</p>
  </blockquote>
  <p>Off the top of my head, only by changing the parameters, you change the hypothesis.</p>
  <blockquote>
  <p>And how can we vary <span class="math-container">$A$</span>?</p>
  </blockquote>
  <p>Let's consider gradient descent as the learning algorithm. In this case, to change the learner, you could change, for example, the learning rate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-26T16:09:37.260Z
Answers.LastActivityDate: 2020-12-07T21:33:50.663Z
Answers.LastEditDate: 2020-12-07T21:33:50.663Z
Answers.UserDisplayName: nbro
Body: <p>What's the distinction between a learning algorithm <span class="math-container">$A$</span> and a hypothesis <span class="math-container">$f$</span>?</p>
  <p>I'm looking for a few concrete examples, if possible.</p>
  <p>For example, would the decision tree and random forest be considered two different learning algorithms? Would a shallow neural network (<strong>that ends up learning a linear function</strong>) and a linear regression model, <strong>both of which use gradient descent to learn parameters</strong>, be considered different learning algorithms?</p>
  <p>Anyway, from what I understand, one way to vary the hypothesis <span class="math-container">$f$</span> would be to change the parameter values, maybe even the hyper-parameter values of, say, a decision tree. Are there other ways of varying <span class="math-container">$f$</span>? And how can we vary <span class="math-container">$A$</span>?</p>
  
Tags: <comparison><terminology><computational-learning-theory><learning-algorithms><hypothesis>
Title: What is the difference between a learning algorithm and a hypothesis?
UserDisplayName: Shirish
ThreadText: # What is the difference between a learning algorithm and a hypothesis?
  
  (Asked by: username_0 on 2019-11-24)
  
  What's the distinction between a learning algorithm $A$ and a hypothesis $f$?
  
  I'm looking for a few concrete examples, if possible.
  
  For example, would the decision tree and random forest be considered two different learning algorithms? Would a shallow neural network (**that ends up learning a linear function**) and a linear regression model, **both of which use gradient descent to learn parameters**, be considered different learning algorithms?
  
  Anyway, from what I understand, one way to vary the hypothesis $f$ would be to change the parameter values, maybe even the hyper-parameter values of, say, a decision tree. Are there other ways of varying $f$? And how can we vary $A$?
  
  **Tags:** comparison, terminology, computational-learning-theory, learning-algorithms, hypothesis
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-11-26. Score: 3 | Accepted answer
  
  In [computational learning theory](http://eliassi.org/COLTSurveyArticle.pdf), a learning algorithm (or learner) $A$ is an algorithm that chooses a hypothesis (which is a function) $h: \mathcal{X} \rightarrow \mathcal{Y}$, where $\mathcal{X}$ is the input space and $\mathcal{Y}$ is the target space, from the hypothesis space $H$.
  
  For example, consider the task of image classification (e.g. [MNIST](http://yann.lecun.com/exdb/mnist/)). You can train, with gradient descent, a neural network to classify the images. In this case, gradient descent is the learner $A$, the space of all possible neural networks that gradient descent considers is the hypothesis space $H$ (so each combination of parameters of the neural network represents a specific hypothesis), $\mathcal{X}$ is the space of images that you want to classify, $\mathcal{Y}$ is the space of all possible classes and the final trained neural network is the hypothesis $h$ chosen by the learner $A$.
  
  > For example, would the decision tree and random forest be considered two different learning algorithms?
  
  The decision tree and random forest are not learning algorithms. A **specific** decision tree or random forest is a **hypothesis** (i.e. function of the form as defined above).
  
  In the context of decision trees, the [ID3 algorithm](https://en.wikipedia.org/wiki/ID3_algorithm) (a [decision tree algorithm](https://en.wikipedia.org/wiki/Decision_tree_learning) that can be used to construct the decision tree, i.e. the hypothesis), is an example of a **learning algorithm** (aka *learner*).
  
  The space of all trees that the learner considers is the **hypothesis space/class**.
  
  > Would a shallow neural network (that ends up learning a linear function) and a linear regression model, both of which use gradient descent to learn parameters, be considered different learning algorithms?
  
  The same can be said here. A specific neural network or linear regression model (i.e. a line) corresponds to a specific hypothesis. The set of all neural networks (or lines, in the case of linear regression) that you consider corresponds to the hypothesis class.
  
  > Anyway, from what I understand, one way to vary the hypothesis $f$ would be to change the parameter values, maybe even the hyper-parameter values of, say, a decision tree.
  
  If you consider a neural network (or decision tree) model, with $N$ parameters $\mathbf{\theta} = [\theta\_i, \dots \theta\_N]$, then a specific combination of these parameters corresponds to a specific hypothesis. If you change the values of these parameters, you also automatically change the hypothesis. If you change the hyperparameters (such as the number of neurons in a specific layer), however, you will be changing the hypothesis class, so the set of hypotheses that you consider.
  
  > Are there other ways of varying $f$?
  
  Off the top of my head, only by changing the parameters, you change the hypothesis.
  
  > And how can we vary $A$?
  
  Let's consider gradient descent as the learning algorithm. In this case, to change the learner, you could change, for example, the learning rate.
  
  
  ---
  
  ### Answer by username_1 on 2019-11-24. Score: 0
  
  A hypothesis is a statement that suggests an as yet unproven explanation of a relationship between two or more phenomena that you intend to test. An agronomist thinks that more nitrogen on canola will always increase the crop output $$Harvest = f(N)$$, or a meteorologist thinks he can show that the path of a hurricane over the ocean can be determined by knowledge of the sea temperature and the wind speed at an altitude of 1000 feet one minute before. $$D(t,0) = f(T(t-1,1000),S(t-1,1000)$$
  Both hypotheses are pegs on which later steps are based; testing follows with a conclusion whether the hypothesis can be rejected or not.
  
  Changing a hypothesis can be simply adding or subtracting arguments to the function or changing the nature of the relationship such as the acceleration of the wind as opposed to its velocity.
  
  A "learning" algorithm describes how the parameters of a numeric model are changed in accordance with the delta rule, that is what the learning rate is and whether momentum is to be applied.
  
  Random Forest and Decision Tree are "classification" algorithms. They are clearly stepwise processes that proceed towards the goal of a model, but they start by specifying the shape that the model will take and place boundaries on what values the parameters may take.
  
  Both learning and classification algorithms specify a priori what shape the model will take and by doing so limit its relevance to particular problems.
  
  
  ---
  
  