Answers.Body: <p>If you want to plot some fitted line, you have to project the hyperplane (your overall fitted model) onto different slices (2D planes), which will indeed produce lines.</p>
  <p>To find the equation of the line, you just need to inject the constants values for each feature into the global model
  <span class="math-container">$$ y = {a_1}{x_1}+{a_2}{x_2}+{a_3}{x_3} + ... + b $$</span></p>
  <p>For example, if you want to observe the best fitted line on a plot <span class="math-container">$y$</span> vs <span class="math-container">$x_1$</span>, you can choose the slice <span class="math-container">$x_2=x_3=...=x_n=0$</span> . This corresponds to the line equation <span class="math-container">$y=a_1x_1+b$</span>.</p>
  <p>Also if you look at a different slice <span class="math-container">$x_2=x_3=...=x_n=1$</span>, then the new line equation is <span class="math-container">$y=a_1x_1+b+\sum_{i=2}^na_i$</span> .</p>
  <p>This also shows that the overall slope will not change (here it's still <span class="math-container">$a_1$</span> in both examples) for slices corresponding to the same feature but the intercept will be affected by the slice choice.</p>
  <p>However you need to be cautious about how you interpret these plots as they only show you a partial view of the full model !</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-12T07:49:16.840Z
Answers.Text: Hyperplanes works best till 3 or 4 features maybe. i don't know how does this plot and best fit line works when it comes to Multiple variables against y_pred. So , are you saying that for 3 features f = x[:0], x[:1], x[:2], i need to use those slices in this equation = y = mx + b , where x is each x[:0] etc. right ? I have somehow plotted x[:0], y_pred. Iam unable to draw conclusions out of it.
Answers.UserDisplayName: Niranjanadas M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-12T10:56:50.197Z
Answers.Text: You just cant visualise the full model if you have more than two features.
Answers.UserDisplayName: Lelouch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-11T12:03:00.270Z
Answers.LastActivityDate: 2023-07-11T12:03:00.270Z
Answers.UserDisplayName: Lelouch
Answers.Body: <pre><code>Isn't the best fit line supposed to be a single line in between the data points
  </code></pre>
  <p>Not here. The linear relationship is between prediction <code>y</code> and each feature <span class="math-container">$x_i$</span> respectively, <strong>while keeping other features constant</strong>. So there is no guarantee that the relationship is linear between predictions.</p>
  <p>Recall for simple linear regression: <span class="math-container">$y = ax+b$</span>, so <span class="math-container">$x$</span> and <span class="math-container">$y$</span> hold a linear relationship.</p>
  <p>But for multivariate linear regression: <span class="math-container">$y = {a_1}{x_1}+{a_2}{x_2}+{a_3}{x_3} + ... + b$</span></p>
  <p>If you think about the relationship between <span class="math-container">$y$</span> and each of the <span class="math-container">$x_i$</span>, it is clear that in general their relationship is linear <em>only when the other <span class="math-container">$x_j$</span> are constant</em>, which is usually not the case.</p>
  <p>Under the same argument, the outputted predictions <span class="math-container">$y$</span> are in general nonlinear too.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T00:23:04.067Z
Answers.Text: Can you explain a bit further. But I am trying to fit best fit line through features and y_pred, right ? I understand that the regressed points are good and close, but i want to see best fit line in single plot or multiple like i plotted (x_train[:, 0], (x_train[:, 1] ...... like so on. Can you explain further a bit more.
Answers.UserDisplayName: Niranjanadas M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-09T05:57:04.840Z
Answers.Text: @NiranjanadasM No, the linear plot you want does not make sense. For 2+ variables it is not a line, but a hyperplane. To visually inspect the prediction result, check e.g. the partial residual plot.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-12T07:43:26.867Z
Answers.Text: Guess I have to study more on hyperplane and how does it effects the plotting. plotting more than 1 feature won't get me a straight best fit line against y_pred. I have to look onto it. Thanks tho for the explanation
Answers.UserDisplayName: Niranjanadas M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T01:32:39.227Z
Answers.LastActivityDate: 2023-07-09T05:48:08.103Z
Answers.LastEditDate: 2023-07-09T05:48:08.103Z
Answers.UserDisplayName: lpounng
Body: <p>I am working on Multiple Linear Regression (Multiple variables). I am been able to predict and get a good r2 score. But I am not sure that I understood the part of plotting the best fit line, I can't understand the visualization of such a model.
  As I am using 50 startups dataset, which have 3 features and 1 target variable (Profit).</p>
  <blockquote>
  <p>The three features and other dataset info from Kaggle(Advertising
  dataset) :
  <a href="https://www.kaggle.com/datasets/yasserh/advertising-sales-dataset" rel="nofollow noreferrer">https://www.kaggle.com/datasets/yasserh/advertising-sales-dataset</a></p>
  </blockquote>
  <p>I will show some of my main code snippets to better illustrate my model.</p>
  <pre><code>main.py 
  
  
  df =  pd.read_csv('ad_sales.csv')
  
  # Perform feature scaling
  # Separate features and target variable
  features = df[['TV Ad Budget (<span class="math-container">$)', 'Radio Ad Budget ($</span>)', 'Newspaper Ad Budget (<span class="math-container">$)']]
  target = df['Sales ($</span>)']
  
  # Perform feature scaling
  scaler = MinMaxScaler()
  scaled_features = scaler.fit_transform(features)
  
  # Create a new DataFrame with scaled features
  features = pd.DataFrame(scaled_features, columns=features.columns)
  
  features = np.array(features)
  labels = np.array(target)
  ## all features are Min-Max Scaled
  
  model = MultipleLinearRegression()  
  
  model.train(x_train = features, y_train = labels, epochs=150)
  
  </code></pre>
  <pre><code>My model : 
  
      def predict(self,x):
          y_pred = [self.b0 + ( (self.b1 * xi[0]) + (self.b2 * xi[1]) + (self.b3 * xi[2]) ) for xi in x]
          return y_pred
          
      def train(self, x_train, y_train, epochs):
          for epoch in range(epochs):
  
              y_pred = self.predict(x_train)
              loss = mse(y_pred, y_train)
              self.optimize(x_train, y_pred, y_train, learning_rate=0.001)
  
          r2 = r2_score(y_train, y_pred)
          print(&quot;R2 Score:&quot;, r2)
  
          # # Scatter plot - Target vs Each Feature
          plt.scatter(x_train[:, 0], y_train, color=&quot;blue&quot;, label=&quot;Actual feature 1&quot;)
          plt.scatter(x_train[:, 0], y_pred, color=&quot;red&quot;, label=&quot;Predicted feature 1&quot;)
          # Line of best fit
          plt.plot(x_train[:, 0], y_pred, color=&quot;green&quot;, linewidth=2, label=&quot;Best fit line&quot;)
          plt.xlabel(&quot;Feature 1&quot;)
          plt.ylabel(&quot;Target&quot;)
          plt.legend()
          plt.show()
  
          plt.scatter(x_train[:, 1], y_train, color=&quot;blue&quot;, label=&quot;Actual feature 2&quot;)
          plt.scatter(x_train[:, 1], y_pred, color=&quot;red&quot;, label=&quot;Predicted feature 2&quot;)
          # Line of best fit
          plt.plot(x_train[:, 1], y_pred, color=&quot;green&quot;, linewidth=2, label=&quot;Best fit line&quot;)
          plt.xlabel(&quot;Feature 2&quot;)
          plt.ylabel(&quot;Target&quot;)
          plt.legend()
          plt.show()
  
          plt.scatter(x_train[:, 2], y_train, color=&quot;blue&quot;, label=&quot;Actual fetaure 3&quot;)
          plt.scatter(x_train[:, 2], y_pred, color=&quot;red&quot;, label=&quot;Predicted feature 3&quot;)
          # Line of best fit
          plt.plot(x_train[:, 2], y_pred, color=&quot;green&quot;, linewidth=2, label=&quot;Best fit line&quot;)
          plt.xlabel(&quot;Feature 3&quot;)
          plt.ylabel(&quot;Target&quot;)
          plt.legend()
          plt.show()
     
    ## As it have 3 features, instead of 3d plot, I am plotting one feature with true labels and y_predicted. the plot screenshot is shared in attachment.
  
  </code></pre>
  <p><a href="https://i.sstatic.net/V7GVy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/V7GVy.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/95nIe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/95nIe.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/jwEgA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jwEgA.png" alt="enter image description here" /></a></p>
  <p>As you can the plot is not showing a straight line, it have a many straight lines combined to match all data points, how ? Isn't the best fit line supposed to be a single line in between the data points . Or it is because of some visualization techniques I used wrong ?</p>
  <p>|| Edited : I want to visualize my Features and if possible want to fit a bit fit line, or curve line if in Poly Model. Like in this Figure :
  <a href="https://i.sstatic.net/25xGC.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/25xGC.jpg" alt="enter image description here" /></a>
  <a href="https://www.megatrend.com/wp-content/uploads/2022/10/2-1.jpg" rel="nofollow noreferrer">https://www.megatrend.com/wp-content/uploads/2022/10/2-1.jpg</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-14T23:11:08.383Z
Comments.Text: Can you upload your 3 feature input raw files? Apparently not even needing to dig deeper into your own optimize() algo details, your pyplot is plotting based on raw (unsorted) input feature slice zipped with their corresponding target labels (noticing the red-only  connecting lines).
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-15T02:13:49.943Z
Comments.Text: Yeah sure, I have edited and added the features link from Kaggle, all those first three are my features ['TV Ad Budget ($)', 'Radio Ad Budget ($)', 'Newspaper Ad Budget ($)'] . @mohottnad
Comments.UserDisplayName: Niranjanadas M
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-16T01:26:49.043Z
Comments.Text: So indeed you are just plotting the raw unordered zipped $(x_i, y)$ tuples, while you seem to want to draw the resultant regression lines for the 3 pairs. Multivariable (homoscedastic) linear regression is a straightforward model with analytics closed form solution for all the coefficients as $\hat{\beta}=(X^TX)^{-1}X^Ty$, or you can use simple gradient descent and even sciki-learn's LinearRegression model.
Comments.UserDisplayName: cinch
Tags: <machine-learning><linear-regression><linear-algebra>
Title: Why my best fit line is not having a single straight line | Multiple Linear Regression
UserDisplayName: Niranjanadas M
ThreadText: # Why my best fit line is not having a single straight line | Multiple Linear Regression
  
  (Asked by: username_0 on 2023-07-07)
  
  I am working on Multiple Linear Regression (Multiple variables). I am been able to predict and get a good r2 score. But I am not sure that I understood the part of plotting the best fit line, I can't understand the visualization of such a model.
  As I am using 50 startups dataset, which have 3 features and 1 target variable (Profit).
  
  > The three features and other dataset info from Kaggle(Advertising
  > dataset) :
  > <https://www.kaggle.com/datasets/yasserh/advertising-sales-dataset>
  
  I will show some of my main code snippets to better illustrate my model.
  
  ```
  main.py 
  
  df =  pd.read_csv('ad_sales.csv')
  
  # Perform feature scaling
  # Separate features and target variable
  features = df[['TV Ad Budget ($)', 'Radio Ad Budget ($)', 'Newspaper Ad Budget ($)']]
  target = df['Sales ($)']
  
  # Perform feature scaling
  scaler = MinMaxScaler()
  scaled_features = scaler.fit_transform(features)
  
  # Create a new DataFrame with scaled features
  features = pd.DataFrame(scaled_features, columns=features.columns)
  
  features = np.array(features)
  labels = np.array(target)
  ## all features are Min-Max Scaled
  
  model = MultipleLinearRegression()  
  
  model.train(x_train = features, y_train = labels, epochs=150)
  
  ```
  
  ```
  My model : 
  
      def predict(self,x):
          y_pred = [self.b0 + ( (self.b1 * xi[0]) + (self.b2 * xi[1]) + (self.b3 * xi[2]) ) for xi in x]
          return y_pred
          
      def train(self, x_train, y_train, epochs):
          for epoch in range(epochs):
  
              y_pred = self.predict(x_train)
              loss = mse(y_pred, y_train)
              self.optimize(x_train, y_pred, y_train, learning_rate=0.001)
  
          r2 = r2_score(y_train, y_pred)
          print("R2 Score:", r2)
  
          # # Scatter plot - Target vs Each Feature
          plt.scatter(x_train[:, 0], y_train, color="blue", label="Actual feature 1")
          plt.scatter(x_train[:, 0], y_pred, color="red", label="Predicted feature 1")
          # Line of best fit
          plt.plot(x_train[:, 0], y_pred, color="green", linewidth=2, label="Best fit line")
          plt.xlabel("Feature 1")
          plt.ylabel("Target")
          plt.legend()
          plt.show()
  
          plt.scatter(x_train[:, 1], y_train, color="blue", label="Actual feature 2")
          plt.scatter(x_train[:, 1], y_pred, color="red", label="Predicted feature 2")
          # Line of best fit
          plt.plot(x_train[:, 1], y_pred, color="green", linewidth=2, label="Best fit line")
          plt.xlabel("Feature 2")
          plt.ylabel("Target")
          plt.legend()
          plt.show()
  
          plt.scatter(x_train[:, 2], y_train, color="blue", label="Actual fetaure 3")
          plt.scatter(x_train[:, 2], y_pred, color="red", label="Predicted feature 3")
          # Line of best fit
          plt.plot(x_train[:, 2], y_pred, color="green", linewidth=2, label="Best fit line")
          plt.xlabel("Feature 3")
          plt.ylabel("Target")
          plt.legend()
          plt.show()
     
    ## As it have 3 features, instead of 3d plot, I am plotting one feature with true labels and y_predicted. the plot screenshot is shared in attachment.
  
  ```
  
  [![enter image description here](https://i.sstatic.net/V7GVy.png)](https://i.sstatic.net/V7GVy.png)
  [![enter image description here](https://i.sstatic.net/95nIe.png)](https://i.sstatic.net/95nIe.png)
  [![enter image description here](https://i.sstatic.net/jwEgA.png)](https://i.sstatic.net/jwEgA.png)
  
  As you can the plot is not showing a straight line, it have a many straight lines combined to match all data points, how ? Isn't the best fit line supposed to be a single line in between the data points . Or it is because of some visualization techniques I used wrong ?
  
  || Edited : I want to visualize my Features and if possible want to fit a bit fit line, or curve line if in Poly Model. Like in this Figure :
  [![enter image description here](https://i.sstatic.net/25xGC.jpg)](https://i.sstatic.net/25xGC.jpg)
  <https://www.megatrend.com/wp-content/uploads/2022/10/2-1.jpg>
  
  **Tags:** machine-learning, linear-regression, linear-algebra
  
  > **Comments:**
  > **username_1**: Can you upload your 3 feature input raw files? Apparently not even needing to dig deeper into your own optimize() algo details, your pyplot is plotting based on raw (unsorted) input feature slice zipped with their corresponding target labels (noticing the red-only  connecting lines).  
  > **username_0**: Yeah sure, I have edited and added the features link from Kaggle, all those first three are my features ['TV Ad Budget ($)', 'Radio Ad Budget ($)', 'Newspaper Ad Budget ($)'] . @mohottnad  
  > **username_1**: So indeed you are just plotting the raw unordered zipped $(x_i, y)$ tuples, while you seem to want to draw the resultant regression lines for the 3 pairs. Multivariable (homoscedastic) linear regression is a straightforward model with analytics closed form solution for all the coefficients as $\hat{\beta}=(X^TX)^{-1}X^Ty$, or you can use simple gradient descent and even sciki-learn's LinearRegression model.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-11. Score: 1
  
  If you want to plot some fitted line, you have to project the hyperplane (your overall fitted model) onto different slices (2D planes), which will indeed produce lines.
  
  To find the equation of the line, you just need to inject the constants values for each feature into the global model
  $$ y = {a\_1}{x\_1}+{a\_2}{x\_2}+{a\_3}{x\_3} + ... + b $$
  
  For example, if you want to observe the best fitted line on a plot $y$ vs $x\_1$, you can choose the slice $x\_2=x\_3=...=x\_n=0$ . This corresponds to the line equation $y=a\_1x\_1+b$.
  
  Also if you look at a different slice $x\_2=x\_3=...=x\_n=1$, then the new line equation is $y=a\_1x\_1+b+\sum\_{i=2}^na\_i$ .
  
  This also shows that the overall slope will not change (here it's still $a\_1$ in both examples) for slices corresponding to the same feature but the intercept will be affected by the slice choice.
  
  However you need to be cautious about how you interpret these plots as they only show you a partial view of the full model !
  
  > **Comments:**
  > **username_0**: Hyperplanes works best till 3 or 4 features maybe. i don't know how does this plot and best fit line works when it comes to Multiple variables against y_pred. So , are you saying that for 3 features f = x[:0], x[:1], x[:2], i need to use those slices in this equation = y = mx + b , where x is each x[:0] etc. right ? I have somehow plotted x[:0], y_pred. Iam unable to draw conclusions out of it.  
  > **username_2**: You just cant visualise the full model if you have more than two features.  
  
  ---
  
  ### Answer by username_3 on 2023-07-07. Score: 1
  
  ```
  Isn't the best fit line supposed to be a single line in between the data points
  
  ```
  
  Not here. The linear relationship is between prediction `y` and each feature $x\_i$ respectively, **while keeping other features constant**. So there is no guarantee that the relationship is linear between predictions.
  
  Recall for simple linear regression: $y = ax+b$, so $x$ and $y$ hold a linear relationship.
  
  But for multivariate linear regression: $y = {a\_1}{x\_1}+{a\_2}{x\_2}+{a\_3}{x\_3} + ... + b$
  
  If you think about the relationship between $y$ and each of the $x\_i$, it is clear that in general their relationship is linear *only when the other $x\_j$ are constant*, which is usually not the case.
  
  Under the same argument, the outputted predictions $y$ are in general nonlinear too.
  
  > **Comments:**
  > **username_0**: Can you explain a bit further. But I am trying to fit best fit line through features and y_pred, right ? I understand that the regressed points are good and close, but i want to see best fit line in single plot or multiple like i plotted (x_train[:, 0], (x_train[:, 1] ...... like so on. Can you explain further a bit more.  
  > **username_3**: @username_0 No, the linear plot you want does not make sense. For 2+ variables it is not a line, but a hyperplane. To visually inspect the prediction result, check e.g. the partial residual plot.  
  > **username_0**: Guess I have to study more on hyperplane and how does it effects the plotting. plotting more than 1 feature won't get me a straight best fit line against y_pred. I have to look onto it. Thanks tho for the explanation  
  
  ---
  
  