Answers.Body: <p>I report three definitions of machine learning (ML) and I also explain that ML can be divided into multiple sub-tasks or sub-categories in <a href="https://ai.stackexchange.com/q/12558/2444">this answer</a>. However, it may not always be clear why classification, regression, or clustering can be considered machine learning tasks or can be solved with ML algorithms/programs, so let me explain why these tasks can be solved with ML, based on Tom Mitchell's definition of an ML algorithm that I report below for completeness.</p>
  <blockquote>
  <p>A computer program is said to <em>learn</em> from <strong>experience</strong> <span class="math-container">$E$</span> with respect to some <strong>class of tasks</strong> <span class="math-container">$T$</span> and <strong>performance measure</strong> <span class="math-container">$P$</span>, if its performance at tasks in <span class="math-container">$T$</span>, as measured by <span class="math-container">$P$</span>, improves with experience <span class="math-container">$E$</span>.</p>
  </blockquote>
  <p>So, according to this definition, for a computer program to be a &quot;machine learner&quot;, we need to identify <span class="math-container">$E$</span>, <span class="math-container">$T$</span>, and <span class="math-container">$P$</span>, and show that the computer program improves with <span class="math-container">$E$</span> at performing the task(s) <span class="math-container">$T$</span>, according to <span class="math-container">$P$</span>.</p>
  <p>For the case of <strong>classification</strong> or <strong>regression</strong> (these are the <em>tasks</em>, which is just a synonym for <em>problems</em>), let's suppose that we are given the training <strong>labelled</strong> dataset <span class="math-container">$D = \{(x_1, y_i), \dots, (x_N, y_N) \}$</span> of <span class="math-container">$N$</span> tuples <span class="math-container">$(x_i, y_i)$</span>, where <span class="math-container">$x_i$</span> are the inputs to the program (or model) and <span class="math-container">$y_i$</span> (the <em>label</em>, aka <em>class</em>, hence the name <em>classification</em>) is the output that the program should produce. In the case of classification, <span class="math-container">$y_i$</span> is an element of a discrete set, e.g. <span class="math-container">$\{0, 1\}$</span> in the case of binary classification, while, in the case of regression, <span class="math-container">$y_i \in \mathbb{R}$</span> is a real number.</p>
  <p>So, in this case, the experience <span class="math-container">$E$</span> is <span class="math-container">$D$</span> (the training labeled dataset). The task <span class="math-container">$T$</span> is <strong>classification</strong> or <strong>regression</strong>, depending on whether <span class="math-container">$y_i$</span> belongs to a discrete or continuous space. The performance measure <span class="math-container">$P$</span> can be e.g. the <strong>cross-entropy</strong> (which is typically used to solve the classification task) or the <strong>mean squared error</strong> (which is typically used to solve the regression task). (I will not recall the definitions of these performance measures here: you can take a book on ML for the details).</p>
  <p>So, we have identified <span class="math-container">$E$</span>, <span class="math-container">$T$</span> and <span class="math-container">$P$</span>. Now, we also need to argue why a program's performance measured by <span class="math-container">$P$</span> would improve with <span class="math-container">$E$</span>. Let's suppose that you are using a neural network trained with gradient descent to solve these tasks. After one epoch/iteration of gradient descent, the loss will be smaller, so the performance of the program will be higher. So, this would indeed be a machine learner, according to Mitchell's definition of ML.</p>
  <p>In the case of <strong>clustering</strong> (the task <span class="math-container">$T$</span>), the only difference is that the dataset <span class="math-container">$D$</span> (i.e. the experience <span class="math-container">$E$</span>) is unlabelled, i.e. we do not have labels, but we are given only the inputs, and the goal, as you know, is to group these inputs based on their similarity, according to some notion of similarity, which is your performance measure <span class="math-container">$P$</span>.</p>
  <p>The other definitions of ML that I report in <a href="https://ai.stackexchange.com/q/12558/2444">the other answer</a> are also consistent with Mitchell's definition. More precisely, most of these definitions are based on the idea that an ML algorithm is an algorithm that &quot;finds patterns in data&quot;. To solve the classification, regression and clustering tasks/problems, an ML algorithm/program needs to find patterns in the data (either explicitly, like in the case of clustering, or indirectly, like in the case of classification), in order for the program's performance to improve.</p>
  <p>Moreover, given that you brought this up in the now-deleted comments, I don't think that the definition of machine learning necessarily implies <strong>prediction</strong> (i.e. the use of a model to forecast something about future data), but, yes, ML is often used for prediction.</p>
  <p>It's also important to note that, when someone says &quot;machine learning&quot; without further details or information (like you did in your post/question), one may refer to</p>
  <ol>
  <li>the field/area of study that studies and applies ML algorithms, or</li>
  <li>an ML algorithm (and sometimes model).</li>
  </ol>
  <p>Mitchell's definition is the definition of what it means for an algorithm/program to be called an ML algorithm/program. However, my definition of ML as a field immediately follows from the definition of an ML algorithm.</p>
  <p>Finally, we could say that a problem that can be formulated in such a way that an ML algorithm can be applied is an ML problem. Note that Mitchell's definition does not mention &quot;ML problem&quot; anywhere, but just &quot;problem&quot;, so this is not a circular definition. So, we could say that classification, regression, and clustering are &quot;ML problems&quot; because they can be solved with ML algorithms. More generally, all these three problems can be thought of as &quot;function approximation&quot; problems, i.e., in all these cases, the solutions are functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T12:55:52.443Z
Answers.Text: This assumes asymptotic learning. This approach agrees with the Statistical Learning Theory, which expects training set to go to infinity. But in the particular ML problems as described in textbooks, as well as in real life, the training set is fixed. It is not increasing. So, it does not make sense for me to define ML problem through increasing training set. Besides, I do not see how clustering fits here: will the performance of clustering algorithms improve with increase of training set? I doubt it.
Answers.UserDisplayName: Marina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T13:02:46.350Z
Answers.Text: We do find patterns in data, but we find very specific patterns. For example, we do not look for sequences of identical numbers. So, to understand what patterns the definition has in mind, one needs to know what ML is.
Answers.UserDisplayName: Marina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T13:25:56.040Z
Answers.Text: @Marina Why do you think that this assumes asymptotic learning? When one says "it improves with experience", one is **not specifying** exactly how or for how long, the only requirement is that it improves with experience at some point. It doesn't always have to improve with experience. In fact, if the learner is already optimal, you don't need more experience, so you can stop learning. So, yes, the training dataset can be fixed and finite, and this is true in almost all cases, and it's always true in practice that the dataset is finite.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T13:28:29.687Z
Answers.Text: I've just explained how clustering fits into this framework. As long as it improves with data/experience according to some performance measure, then it's a machine learning algorithm/program (or machine learner). E.g., you can take [k-means](https://en.wikipedia.org/wiki/K-means_clustering). In k-means, we want to minimize a certain objective function. That objective function is the performance measure and, if k-means finds a local optimum of that objective function starting from some randomly located point in the search space, then it's improved with experience, so k-means is an ML algorithm.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T13:30:03.720Z
Answers.Text: Your definition says: IF experience increases THEN the performance increases. It says nothing about the case when experience does not increase.
Answers.UserDisplayName: Marina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T13:31:52.763Z
Answers.Text: Your definition requires to define the class of tasks T. Is not it the same as define ML?
Answers.UserDisplayName: Marina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T13:32:06.987Z
Answers.Text: @Marina That's not my definition of machine learning, but [Tom Mitchell](http://www.cs.cmu.edu/~tom/)'s definition, which I think is good enough to describe most ML algorithms. Yes, it's true that it doesn't say anything about when the experience does not increase, but the point/idea of this definition is to define what it means "for a program to learn". If a program respects what is written in that definition, then it's learning, i.e. it's an ML program, otherwise, it's not.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T13:33:29.107Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/120417/discussion-between-marina-and-nbro).
Answers.UserDisplayName: Marina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T16:16:54.210Z
Answers.Text: You give example of defining tasks of classification or regression, then you give examples of criteria. Then you say " So, we have identified E, T and P. " No, we have not.
Answers.UserDisplayName: Marina
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T16:17:48.320Z
Answers.Text: @Marina Can you explain why not? I'm saying what they are in my answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-03T15:58:40.000Z
Answers.Text: @nbro In the definition should the word "improves" interpreted literally? I mean we can have an algorithm that becomes worse (as measured by P) with experience. So is it correct to say that"improves" give the idea of learning? It is like in school, if you are doing good at exams it should be the case that you have learned something in order to be able to get good grades, otherwise not.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-03T16:10:55.023Z
Answers.Text: @adosar If your performance doesn't improve, then have you learned with respect to that performance measure or specific goal? I don't think so. Maybe you learned other things, with respect to other vaguely defined performance measures or goals. That's maybe what confuses you. For instance, if your goal is to learn quantum mechanics, but your performance at the exercises never improves, then you have not learned anything, unless you're making mistakes on purpose, but that's a different story. Maybe you learned about general relativity, but that may not help you with the QM exam.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-03T17:48:39.640Z
Answers.Text: @nbro Thats what I wanted to say but you stated it and made it more clear.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-03T22:24:21.587Z
Answers.Text: @nbro Thank you for this answer; it really helped me understand Mitchell's definition (especially how to interpret "Experience").  May I ask a clarification question?  In the case of clustering, should I imagine T is "divide this dataset into clusters" and E is "the entire dataset"?  It feels a little circular or vacuous in that case... of course you improve at clustering the dataset using the dataset.  Do you see what I'm confused by?
Answers.UserDisplayName: CJD
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T12:07:08.617Z
Answers.LastActivityDate: 2021-03-04T16:22:45.747Z
Answers.LastEditDate: 2021-03-04T16:22:45.747Z
Answers.UserDisplayName: nbro
Body: <p>How to define machine learning to cover clustering, classification, and regression? What unites these problems?</p>
  
Tags: <machine-learning><classification><definitions><regression><clustering>
Title: How to define machine learning to cover clustering, classification, and regression?
UserDisplayName: Marina
ThreadText: # How to define machine learning to cover clustering, classification, and regression?
  
  (Asked by: username_0 on 2021-03-03)
  
  How to define machine learning to cover clustering, classification, and regression? What unites these problems?
  
  **Tags:** machine-learning, classification, definitions, regression, clustering
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-04. Score: 3
  
  I report three definitions of machine learning (ML) and I also explain that ML can be divided into multiple sub-tasks or sub-categories in [this answer](https://ai.stackexchange.com/q/12558/2444). However, it may not always be clear why classification, regression, or clustering can be considered machine learning tasks or can be solved with ML algorithms/programs, so let me explain why these tasks can be solved with ML, based on Tom Mitchell's definition of an ML algorithm that I report below for completeness.
  
  > A computer program is said to *learn* from **experience** $E$ with respect to some **class of tasks** $T$ and **performance measure** $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.
  
  So, according to this definition, for a computer program to be a "machine learner", we need to identify $E$, $T$, and $P$, and show that the computer program improves with $E$ at performing the task(s) $T$, according to $P$.
  
  For the case of **classification** or **regression** (these are the *tasks*, which is just a synonym for *problems*), let's suppose that we are given the training **labelled** dataset $D = \{(x\_1, y\_i), \dots, (x\_N, y\_N) \}$ of $N$ tuples $(x\_i, y\_i)$, where $x\_i$ are the inputs to the program (or model) and $y\_i$ (the *label*, aka *class*, hence the name *classification*) is the output that the program should produce. In the case of classification, $y\_i$ is an element of a discrete set, e.g. $\{0, 1\}$ in the case of binary classification, while, in the case of regression, $y\_i \in \mathbb{R}$ is a real number.
  
  So, in this case, the experience $E$ is $D$ (the training labeled dataset). The task $T$ is **classification** or **regression**, depending on whether $y\_i$ belongs to a discrete or continuous space. The performance measure $P$ can be e.g. the **cross-entropy** (which is typically used to solve the classification task) or the **mean squared error** (which is typically used to solve the regression task). (I will not recall the definitions of these performance measures here: you can take a book on ML for the details).
  
  So, we have identified $E$, $T$ and $P$. Now, we also need to argue why a program's performance measured by $P$ would improve with $E$. Let's suppose that you are using a neural network trained with gradient descent to solve these tasks. After one epoch/iteration of gradient descent, the loss will be smaller, so the performance of the program will be higher. So, this would indeed be a machine learner, according to Mitchell's definition of ML.
  
  In the case of **clustering** (the task $T$), the only difference is that the dataset $D$ (i.e. the experience $E$) is unlabelled, i.e. we do not have labels, but we are given only the inputs, and the goal, as you know, is to group these inputs based on their similarity, according to some notion of similarity, which is your performance measure $P$.
  
  The other definitions of ML that I report in [the other answer](https://ai.stackexchange.com/q/12558/2444) are also consistent with Mitchell's definition. More precisely, most of these definitions are based on the idea that an ML algorithm is an algorithm that "finds patterns in data". To solve the classification, regression and clustering tasks/problems, an ML algorithm/program needs to find patterns in the data (either explicitly, like in the case of clustering, or indirectly, like in the case of classification), in order for the program's performance to improve.
  
  Moreover, given that you brought this up in the now-deleted comments, I don't think that the definition of machine learning necessarily implies **prediction** (i.e. the use of a model to forecast something about future data), but, yes, ML is often used for prediction.
  
  It's also important to note that, when someone says "machine learning" without further details or information (like you did in your post/question), one may refer to
  
  1. the field/area of study that studies and applies ML algorithms, or
  2. an ML algorithm (and sometimes model).
  
  Mitchell's definition is the definition of what it means for an algorithm/program to be called an ML algorithm/program. However, my definition of ML as a field immediately follows from the definition of an ML algorithm.
  
  Finally, we could say that a problem that can be formulated in such a way that an ML algorithm can be applied is an ML problem. Note that Mitchell's definition does not mention "ML problem" anywhere, but just "problem", so this is not a circular definition. So, we could say that classification, regression, and clustering are "ML problems" because they can be solved with ML algorithms. More generally, all these three problems can be thought of as "function approximation" problems, i.e., in all these cases, the solutions are functions.
  
  > **Comments:**
  > **username_0**: This assumes asymptotic learning. This approach agrees with the Statistical Learning Theory, which expects training set to go to infinity. But in the particular ML problems as described in textbooks, as well as in real life, the training set is fixed. It is not increasing. So, it does not make sense for me to define ML problem through increasing training set. Besides, I do not see how clustering fits here: will the performance of clustering algorithms improve with increase of training set? I doubt it.  
  > **username_0**: We do find patterns in data, but we find very specific patterns. For example, we do not look for sequences of identical numbers. So, to understand what patterns the definition has in mind, one needs to know what ML is.  
  > **username_1**: @Marina Why do you think that this assumes asymptotic learning? When one says "it improves with experience", one is **not specifying** exactly how or for how long, the only requirement is that it improves with experience at some point. It doesn't always have to improve with experience. In fact, if the learner is already optimal, you don't need more experience, so you can stop learning. So, yes, the training dataset can be fixed and finite, and this is true in almost all cases, and it's always true in practice that the dataset is finite.  
  > **username_1**: I've just explained how clustering fits into this framework. As long as it improves with data/experience according to some performance measure, then it's a machine learning algorithm/program (or machine learner). E.g., you can take [k-means](https://en.wikipedia.org/wiki/K-means_clustering). In k-means, we want to minimize a certain objective function. That objective function is the performance measure and, if k-means finds a local optimum of that objective function starting from some randomly located point in the search space, then it's improved with experience, so k-means is an ML algorithm.  
  > **username_0**: Your definition says: IF experience increases THEN the performance increases. It says nothing about the case when experience does not increase.  
  > **username_0**: Your definition requires to define the class of tasks T. Is not it the same as define ML?  
  > **username_1**: @Marina That's not my definition of machine learning, but [Tom Mitchell](http://www.cs.cmu.edu/~tom/)'s definition, which I think is good enough to describe most ML algorithms. Yes, it's true that it doesn't say anything about when the experience does not increase, but the point/idea of this definition is to define what it means "for a program to learn". If a program respects what is written in that definition, then it's learning, i.e. it's an ML program, otherwise, it's not.  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/120417/discussion-between-marina-and-nbro).  
  > **username_0**: You give example of defining tasks of classification or regression, then you give examples of criteria. Then you say " So, we have identified E, T and P. " No, we have not.  
  > **username_1**: @Marina Can you explain why not? I'm saying what they are in my answer.  
  > **username_2**: @nbro In the definition should the word "improves" interpreted literally? I mean we can have an algorithm that becomes worse (as measured by P) with experience. So is it correct to say that"improves" give the idea of learning? It is like in school, if you are doing good at exams it should be the case that you have learned something in order to be able to get good grades, otherwise not.  
  > **username_1**: @adosar If your performance doesn't improve, then have you learned with respect to that performance measure or specific goal? I don't think so. Maybe you learned other things, with respect to other vaguely defined performance measures or goals. That's maybe what confuses you. For instance, if your goal is to learn quantum mechanics, but your performance at the exercises never improves, then you have not learned anything, unless you're making mistakes on purpose, but that's a different story. Maybe you learned about general relativity, but that may not help you with the QM exam.  
  > **username_2**: @nbro Thats what I wanted to say but you stated it and made it more clear.  
  > **username_3**: @nbro Thank you for this answer; it really helped me understand Mitchell's definition (especially how to interpret "Experience").  May I ask a clarification question?  In the case of clustering, should I imagine T is "divide this dataset into clusters" and E is "the entire dataset"?  It feels a little circular or vacuous in that case... of course you improve at clustering the dataset using the dataset.  Do you see what I'm confused by?  
  
  ---
  
  