Answers.Body: <p>The <code>cross_val_score</code> function of the sci-kit-learn library uses the test dataset's mean to calculate the R2 score during each cross-validation fold. This is not called cheating since the R2 score is designed to evaluate how well the model is performing generalized on new unseen datasets and using the test set's mean is a standard part of the R² calculation.</p>
  <p>you are right about it y_true.mean is from the testing dataset but it is not cheating because R2 &gt; 0 means the model performs better than a constant baseline that predicts the mean of the test dataset.</p>
  <p>If your model can't beat this simple baseline, it means the model isn't effectively learning the relationship between the input features and the target.</p>
  <p>The baseline comparison is not meant to &quot;cheat&quot; by using information from the testing set; rather, it is a fair baseline that checks if your model is doing better than simply predicting the average of the testing set’s target values.</p>
  <h3>Should R² &gt; -0.01 Be the Baseline?</h3>
  <p>Not really. The baseline you are competing against is R² = 0, which is the score of a constant model that predicts the mean of y_test. A score below 0 means the model is performing worse than simply predicting the mean of the test data, but R² &gt; 0 is the standard for doing better than a baseline model.</p>
  <p>I am not going much deep into the interpretation of the r2 score like what the value of r2 suggests about the model results since <code>Luca Anzalone</code> has explained well in above answer</p>
  <p>for the question So doing better than a baseline wouldn't be having R2&gt;0
  but rather R2&gt;−0.01
  or something like this?</p>
  <p>=&gt; No, doing better than the baseline is indicated by having an R² score greater than 0. it is due to following reasons</p>
  <p>An R² score of 0 means that your model performs as well as a constant model that predicts the mean of the target values from the test set, ignoring the input features.</p>
  <p>An R² score greater than 0 means your model is doing better than the baseline (constant mean prediction).</p>
  <p>An R² score less than 0 means your model is doing worse than the baseline.</p>
  <p>If your model had an R² score of something like -0.01, that would imply that the model's predictions are slightly worse than just predicting the mean of the test set. But the true goal is to have R² &gt; 0, which indicates that your model is leveraging the features to make predictions that outperform the simple mean baseline.</p>
  <p>if you have still doubt feel free to ask and also ignore some notation mistakes since i am new at typing notation on stack :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T07:53:58.363Z
Answers.Text: but if my R² is exactly 0 it means I was able to do as well as if I had _access_ to the mean of the test set, _without having access to_ the test set, so I should expect a "pat on the back", not a score of 0, right?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T08:14:39.830Z
Answers.Text: R² = 0 means your model’s predictions are no better than simply predicting the mean of the test set . While it’s true that you didn't "cheat" by using the test set directly, R² = 0 means your model’s predictions are no better than simply predicting the mean of the test set  While it’s true that you didn't "cheat" by using the test set directly since the R² score is designed to answer: "How well does my model use the input features to predict the target variable? R² = 0 doesn't punish you, but it reflects that your model hasn't added value beyond predicting the test mean.
Answers.UserDisplayName: Keval Pandya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T09:15:04.103Z
Answers.Text: Ok, so one should still not be unhappy if R² = 0, it would mean that one has learnt $\epsilon > 0$ about the task at hand with the training set. **I would hence expect that the strategy of 'taking the mean of the output variable of the training set to predict the output variable in the test set' would on average get me R² < 0.** Is this really desirable?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T09:39:45.630Z
Answers.Text: Yes you are right   Achieving R² = 0 does indicate that your model is doing at least as well as predicting the test set’s mean, even though you don't have access to it. That means your model has indeed learned something about the task at hand using the training data. So, in a practical sense, you're not in a bad spot if R² = 0 — it just reflects that you're not yet improving over the baseline using the input features.
Answers.UserDisplayName: Keval Pandya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T09:41:07.633Z
Answers.Text: If you use the strategy of predicting the mean of the output variable from the training set for the test set, you're right to suspect that this could often result in an R² less than 0. because R2 means the generalization of the model its formula compares the model results with the baseline of the mean of the test set R² < 0 indicates that your model is worse than simply predicting the mean of the test set. This happens because your model's training-set-based predictions don't fit the test set as well as the constant mean baseline would
Answers.UserDisplayName: Keval Pandya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T09:44:24.847Z
Answers.Text: for your question  Is this desirable?  In the context of model evaluation, yes, this is desirable — the way R² is structured serves a purpose to encourage the improvement since it is made for the improvement of the generalization of the model and it is also Punishes overfitting or underfitting  R2<0 happens when your model is not performing well on new dataset because of the underfitting and the overfitting, so it gives you hint about overfitting or underfitting.
Answers.UserDisplayName: Keval Pandya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T11:05:53.533Z
Answers.Text: Thanks! can't award bounty just yet but will do when i can
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T11:21:18.927Z
Answers.Text: Ok No issue , if you have still doubt on that feel free to ask :)
Answers.UserDisplayName: Keval Pandya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-24T05:19:46.260Z
Answers.LastActivityDate: 2024-10-24T05:19:46.260Z
Answers.UserDisplayName: Keval Pandya
Answers.Body: <p>Let's clarify this with a numerical example. Let's assume <code>y_true = [0, 1, 1]</code> to be the true class labels of <span class="math-container">$N=3$</span> test points. Therefore, <code>y_true.mean() = 2/3</code>.</p>
  <p><strong>Case 1</strong>: everthing is wrongly predicted, so <code>y_pred = [1, 0, 0]</code>. Let's compute <span class="math-container">$u$</span> and <span class="math-container">$v$</span>.
  <span class="math-container">$$
  \begin{align}
  u &amp;= \sum(y_\text{true}-y_\text{pred})^2 = (0-1)^2+(1-0)^2
  +(1-0)^2 = 3 \\
  v &amp;= \sum \Big(y_\text{true}-\frac23\Big)^2 = (-2/3)^2+(1/3)^2+(1/3)^2 = \frac69 = \frac23
  \end{align}$$</span>
  Now we compute <span class="math-container">$R^2$</span>, so:
  <span class="math-container">$$R^2 = 1 - \frac{u}{v} = 1 - \frac{3}{2/3} = 1 - 2 = \mathbf{-1}$$</span>
  You get -1 because <code>y_pred</code> is always wrong. so you can think of <span class="math-container">$u$</span> as counting the number of errors (it's so if you predict the label to be exactly zero or one), whereas <span class="math-container">$v$</span> is the ratio of positive labels.</p>
  <p><strong>Case 2:</strong> perfect predictions, so <code>y_pred = y_true</code>.
  <span class="math-container">$$
  \begin{align}
  u &amp;= (0-0)^2+(1-1)^2
  +(1-1)^2 = 0\quad\quad\text{(no errors)} \\
  R^2 &amp;= 1 -\frac{0}{2/3} = 1
  \end{align}$$</span>
  <span class="math-container">$R^2=1$</span> because every label is correctly predicted, so there are no errors and the correlation is the most possible.</p>
  <p><strong>Case 3:</strong> some errors, say <code>y_pred=[0, 1, 0]</code>. Again:
  <span class="math-container">$$
  \begin{align}
  u &amp;= (0-0)^2+(1-1)^2
  +(1-0)^2 = 1 \\
  R^2 &amp;= 1 -\frac{1}{2/3} = 1 - \frac32 = -\frac12
  \end{align}$$</span>
  <span class="math-container">$R^2=-0.5$</span>, so negative correlation. From this it seems that <span class="math-container">$R^2$</span>, in general, gives more importance to correctly predicting the positive class.</p>
  <p>So, there is no issue about using the mean of the true labels because that info is not used during training but only to compute an evaluation metric.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-01T22:48:12.717Z
Answers.Text: Maybe my post wasn't clear but I'm talking about `cross_val_score` which evaluates the coefficient of determination on the testing set?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-02T10:12:13.507Z
Answers.Text: I see. `cross_val_score` takes a model and a dataset ($X$, $y$). It trains the model $k$ times, each time leaves out (for validation) a "fold", and then computes the score (e.g. $R^2$) on that fold. You use `cross_val_score` to validate your model, so $(X, y)$ is actually the **training set** not the test set (otherwise you "cheat"). Once you determined the best model you retrain it on all the train-set, and do the final evaluation on test (e.g. by computing $R^2$). Read [here](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) for more.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-02T15:18:02.933Z
Answers.Text: I see. But when computing the score (e.g. $R^2$), it does use `y_true.mean` in `v`, right? `y_true.mean` being the mean of the testing fold.
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-02T18:00:25.317Z
Answers.Text: You perform `cross_val_score` on $(X_\text{train},y_\text{train})$, so for each fold (there are $k$ of them) you have $(X_\text{train}^k,y_\text{train}^k)$ and $(X_\text{val}^k,y_\text{val}^k)$. Evaluation is performed on the latter pair, and `y_true.mean` refers to $y_\text{val}^k$. You do this, see the results and if satisfied you finally evaluate on the test-set: you don't use `cross_val_score` otherwise it would train on test data which is wrong, but instead use [r2_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-03T13:52:04.040Z
Answers.Text: I see, you don't use `cross_val_score` to get the the coefficient of determination of your method. But, to get the coefficient of determination of your method, you calculate it on the test-set. However, I believe my questions still stand. The `y_true.mean` is now $y_{test}$ right? So when calculating the coefficient of determination on the test-set, `v` depends on the mean of the test set. So if $R^2_{test}$ is equal to 0 it means we do as well as a method that inferred a little bit something from the test set (`y_true.mean`) so we learned a little more than just noise?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-01T09:01:26.510Z
Answers.LastActivityDate: 2023-06-01T09:01:26.510Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>The function <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html" rel="nofollow noreferrer">cross_val_score</a> uses the estimator’s default scorer (if available) and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html" rel="nofollow noreferrer">LinearRgression</a> (the estimator I use) uses <em>The coefficient of determination</em> (which  is defined as <span class="math-container">$R^2 = 1 - \frac{u}{v}$</span>, where
  <span class="math-container">$u$</span> is the residual sum of squares <code>((y_true - y_pred)** 2).sum()</code> and
  <span class="math-container">$v$</span> is the total sum of squares <code>((y_true - y_true.mean()) ** 2).sum()</code>.
  The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a score of 0.0.)</p>
  <p>Is <code>y_true.mean</code> the <code>y_true.mean</code> of the training set or the testing set? If it's the one of the testing set, isn't it &quot;cheating&quot; i.e. we compare our predictions to a method that has inferred something from the testing set?</p>
  <p>So doing better than a baseline wouldn't be having <span class="math-container">$R^2 &gt; 0$</span> but rather <span class="math-container">$R^2 &gt; -0.01$</span> or something like this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-31T20:14:21.747Z
Comments.Text: You have `model.score(x_test, y_test)`. Then it computes $R^2$ according to $\hat{y}=model(x_\text{test})$. $u$ accounts for the predictions, whereas $v$ represents an ideal situation in which $\hat{y}$ is the same as `y_test` (i.e. no errors).  You do the ratio of that, so what's the issue?
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-31T21:30:43.837Z
Comments.Text: From my understanding $v$ represents the situation where I predict the mean, not a situation with no errors? (otherwise $v$ would be $0$ and you'd be dividing by $0$)
Comments.UserDisplayName: FluidMechanics Potential Flows
Tags: <linear-regression><cross-validation><scikit-learn>
Title: cross_val_score of sklearn and LinearRegression scoring method
UserDisplayName: FluidMechanics Potential Flows
ThreadText: # cross_val_score of sklearn and LinearRegression scoring method
  
  (Asked by: username_0 on 2023-05-31)
  
  The function [cross\_val\_score](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) uses the estimator’s default scorer (if available) and [LinearRgression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) (the estimator I use) uses *The coefficient of determination* (which is defined as $R^2 = 1 - \frac{u}{v}$, where
  $u$ is the residual sum of squares `((y_true - y_pred)** 2).sum()` and
  $v$ is the total sum of squares `((y_true - y_true.mean()) ** 2).sum()`.
  The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a score of 0.0.)
  
  Is `y_true.mean` the `y_true.mean` of the training set or the testing set? If it's the one of the testing set, isn't it "cheating" i.e. we compare our predictions to a method that has inferred something from the testing set?
  
  So doing better than a baseline wouldn't be having $R^2 > 0$ but rather $R^2 > -0.01$ or something like this?
  
  **Tags:** linear-regression, cross-validation, scikit-learn
  
  > **Comments:**
  > **username_1**: You have `model.score(x_test, y_test)`. Then it computes $R^2$ according to $\hat{y}=model(x_\text{test})$. $u$ accounts for the predictions, whereas $v$ represents an ideal situation in which $\hat{y}$ is the same as `y_test` (i.e. no errors).  You do the ratio of that, so what's the issue?  
  > **username_0**: From my understanding $v$ represents the situation where I predict the mean, not a situation with no errors? (otherwise $v$ would be $0$ and you'd be dividing by $0$)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-10-24. Score: 0 | Accepted answer
  
  The `cross_val_score` function of the sci-kit-learn library uses the test dataset's mean to calculate the R2 score during each cross-validation fold. This is not called cheating since the R2 score is designed to evaluate how well the model is performing generalized on new unseen datasets and using the test set's mean is a standard part of the R² calculation.
  
  you are right about it y\_true.mean is from the testing dataset but it is not cheating because R2 > 0 means the model performs better than a constant baseline that predicts the mean of the test dataset.
  
  If your model can't beat this simple baseline, it means the model isn't effectively learning the relationship between the input features and the target.
  
  The baseline comparison is not meant to "cheat" by using information from the testing set; rather, it is a fair baseline that checks if your model is doing better than simply predicting the average of the testing set’s target values.
  
  ### Should R² > -0.01 Be the Baseline?
  
  Not really. The baseline you are competing against is R² = 0, which is the score of a constant model that predicts the mean of y\_test. A score below 0 means the model is performing worse than simply predicting the mean of the test data, but R² > 0 is the standard for doing better than a baseline model.
  
  I am not going much deep into the interpretation of the r2 score like what the value of r2 suggests about the model results since `username_1 username_1` has explained well in above answer
  
  for the question So doing better than a baseline wouldn't be having R2>0
  but rather R2>−0.01
  or something like this?
  
  => No, doing better than the baseline is indicated by having an R² score greater than 0. it is due to following reasons
  
  An R² score of 0 means that your model performs as well as a constant model that predicts the mean of the target values from the test set, ignoring the input features.
  
  An R² score greater than 0 means your model is doing better than the baseline (constant mean prediction).
  
  An R² score less than 0 means your model is doing worse than the baseline.
  
  If your model had an R² score of something like -0.01, that would imply that the model's predictions are slightly worse than just predicting the mean of the test set. But the true goal is to have R² > 0, which indicates that your model is leveraging the features to make predictions that outperform the simple mean baseline.
  
  if you have still doubt feel free to ask and also ignore some notation mistakes since i am new at typing notation on stack :)
  
  > **Comments:**
  > **username_0**: but if my R² is exactly 0 it means I was able to do as well as if I had _access_ to the mean of the test set, _without having access to_ the test set, so I should expect a "pat on the back", not a score of 0, right?  
  > **username_2**: R² = 0 means your model’s predictions are no better than simply predicting the mean of the test set . While it’s true that you didn't "cheat" by using the test set directly, R² = 0 means your model’s predictions are no better than simply predicting the mean of the test set  While it’s true that you didn't "cheat" by using the test set directly since the R² score is designed to answer: "How well does my model use the input features to predict the target variable? R² = 0 doesn't punish you, but it reflects that your model hasn't added value beyond predicting the test mean.  
  > **username_0**: Ok, so one should still not be unhappy if R² = 0, it would mean that one has learnt $\epsilon > 0$ about the task at hand with the training set. **I would hence expect that the strategy of 'taking the mean of the output variable of the training set to predict the output variable in the test set' would on average get me R² < 0.** Is this really desirable?  
  > **username_2**: Yes you are right   Achieving R² = 0 does indicate that your model is doing at least as well as predicting the test set’s mean, even though you don't have access to it. That means your model has indeed learned something about the task at hand using the training data. So, in a practical sense, you're not in a bad spot if R² = 0 — it just reflects that you're not yet improving over the baseline using the input features.  
  > **username_2**: If you use the strategy of predicting the mean of the output variable from the training set for the test set, you're right to suspect that this could often result in an R² less than 0. because R2 means the generalization of the model its formula compares the model results with the baseline of the mean of the test set R² < 0 indicates that your model is worse than simply predicting the mean of the test set. This happens because your model's training-set-based predictions don't fit the test set as well as the constant mean baseline would  
  > **username_2**: for your question  Is this desirable?  In the context of model evaluation, yes, this is desirable — the way R² is structured serves a purpose to encourage the improvement since it is made for the improvement of the generalization of the model and it is also Punishes overfitting or underfitting  R2<0 happens when your model is not performing well on new dataset because of the underfitting and the overfitting, so it gives you hint about overfitting or underfitting.  
  > **username_0**: Thanks! can't award bounty just yet but will do when i can  
  > **username_2**: Ok No issue , if you have still doubt on that feel free to ask :)  
  
  ---
  
  ### Answer by username_1 on 2023-06-01. Score: 0
  
  Let's clarify this with a numerical example. Let's assume `y_true = [0, 1, 1]` to be the true class labels of $N=3$ test points. Therefore, `y_true.mean() = 2/3`.
  
  **Case 1**: everthing is wrongly predicted, so `y_pred = [1, 0, 0]`. Let's compute $u$ and $v$.
  $$
  \begin{align}
  u &= \sum(y\_\text{true}-y\_\text{pred})^2 = (0-1)^2+(1-0)^2
  +(1-0)^2 = 3 \\
  v &= \sum \Big(y\_\text{true}-\frac23\Big)^2 = (-2/3)^2+(1/3)^2+(1/3)^2 = \frac69 = \frac23
  \end{align}$$
  Now we compute $R^2$, so:
  $$R^2 = 1 - \frac{u}{v} = 1 - \frac{3}{2/3} = 1 - 2 = \mathbf{-1}$$
  You get -1 because `y_pred` is always wrong. so you can think of $u$ as counting the number of errors (it's so if you predict the label to be exactly zero or one), whereas $v$ is the ratio of positive labels.
  
  **Case 2:** perfect predictions, so `y_pred = y_true`.
  $$
  \begin{align}
  u &= (0-0)^2+(1-1)^2
  +(1-1)^2 = 0\quad\quad\text{(no errors)} \\
  R^2 &= 1 -\frac{0}{2/3} = 1
  \end{align}$$
  $R^2=1$ because every label is correctly predicted, so there are no errors and the correlation is the most possible.
  
  **Case 3:** some errors, say `y_pred=[0, 1, 0]`. Again:
  $$
  \begin{align}
  u &= (0-0)^2+(1-1)^2
  +(1-0)^2 = 1 \\
  R^2 &= 1 -\frac{1}{2/3} = 1 - \frac32 = -\frac12
  \end{align}$$
  $R^2=-0.5$, so negative correlation. From this it seems that $R^2$, in general, gives more importance to correctly predicting the positive class.
  
  So, there is no issue about using the mean of the true labels because that info is not used during training but only to compute an evaluation metric.
  
  > **Comments:**
  > **username_0**: Maybe my post wasn't clear but I'm talking about `cross_val_score` which evaluates the coefficient of determination on the testing set?  
  > **username_1**: I see. `cross_val_score` takes a model and a dataset ($X$, $y$). It trains the model $k$ times, each time leaves out (for validation) a "fold", and then computes the score (e.g. $R^2$) on that fold. You use `cross_val_score` to validate your model, so $(X, y)$ is actually the **training set** not the test set (otherwise you "cheat"). Once you determined the best model you retrain it on all the train-set, and do the final evaluation on test (e.g. by computing $R^2$). Read [here](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation) for more.  
  > **username_0**: I see. But when computing the score (e.g. $R^2$), it does use `y_true.mean` in `v`, right? `y_true.mean` being the mean of the testing fold.  
  > **username_1**: You perform `cross_val_score` on $(X_\text{train},y_\text{train})$, so for each fold (there are $k$ of them) you have $(X_\text{train}^k,y_\text{train}^k)$ and $(X_\text{val}^k,y_\text{val}^k)$. Evaluation is performed on the latter pair, and `y_true.mean` refers to $y_\text{val}^k$. You do this, see the results and if satisfied you finally evaluate on the test-set: you don't use `cross_val_score` otherwise it would train on test data which is wrong, but instead use [r2_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score)  
  > **username_0**: I see, you don't use `cross_val_score` to get the the coefficient of determination of your method. But, to get the coefficient of determination of your method, you calculate it on the test-set. However, I believe my questions still stand. The `y_true.mean` is now $y_{test}$ right? So when calculating the coefficient of determination on the test-set, `v` depends on the mean of the test set. So if $R^2_{test}$ is equal to 0 it means we do as well as a method that inferred a little bit something from the test set (`y_true.mean`) so we learned a little more than just noise?  
  
  ---
  
  