Answers.Body: <p>What you mentioned is called <strong>feature engineering</strong>, <em>i.e.</em>, the process of using domain knowledge to create new features or modify existing ones in a way that makes machine learning algorithms work more effectively.</p>
  <p>Given a training set <span class="math-container">$X=\left\{x^{(i)}, y^{(i)} \right\}_{i=1}^{m}$</span>, with <span class="math-container">$x^{(i)} \in \mathbb{R}^{n}$</span> being the <span class="math-container">$i$</span>-th sample, and <span class="math-container">$y^{(i)} \in \mathbb{R}$</span> the outcome, if you think the ratio of two features <span class="math-container">$X_1/X_2$</span> is more relevant to the prediction of <span class="math-container">$y$</span> than <span class="math-container">$X_1$</span> and <span class="math-container">$X_2$</span> separately, then you should compute this new hand-crafted feature, and drop <span class="math-container">$X_1$</span> and <span class="math-container">$X_2$</span> (see avoiding collinearity below).</p>
  <p>However, training a linear regression on <span class="math-container">$X_1$</span> and <span class="math-container">$X_2$</span> versus training on the new feature <span class="math-container">$\frac{X_1}{X_2}$</span> results in fundamentally different models.  The linear regression model is defined by:
  <span class="math-container">$$
  \hat{y}^{(i)} = \vartheta_0 + \vartheta_1 x^{(i)}_1 + \vartheta_2 x^{(i)}_2 + \ldots + \vartheta_n x^{(i)}_n
  $$</span>
  where <span class="math-container">$\hat{y}^{(i)}$</span> is the predicted value for the <span class="math-container">$i$</span>-th sample. By contrast, if we introduce the new feature <span class="math-container">$\frac{x^{(i)}_1}{x^{(i)}_2}$</span>, the model becomes:
  <span class="math-container">$$
  \hat{y}^{(i)} = \vartheta_0 + \vartheta_1 \frac{x^{(i)}_1}{x^{(i)}_2} + \vartheta_3 x^{(i)}_3 + \ldots + \vartheta_n x^{(i)}_n
  $$</span>
  Here, the ratio <span class="math-container">$\frac{X_1}{X_2}$</span> is weighted by a single parameter, <span class="math-container">$\vartheta_1$</span>, instad of <span class="math-container">$\vartheta_1$</span> and <span class="math-container">$\vartheta_2$</span> separetely. This modification changes the model’s architecture (<em>i.e.</em>, the number of trainable parameters <span class="math-container">$\vartheta_k$</span>) and therefore the model's performance.</p>
  <h2>Avoid collinearity</h2>
  <p>A good rule is that of dropping from your training set linearly dependent features. Suppose you have two different features, <span class="math-container">$X_1$</span> and <span class="math-container">$X_2$</span>, that are linearly dependent  (or collinear):
  <span class="math-container">$$X_2 = aX_1 + b$$</span>
  where <span class="math-container">$a, b \in \mathbb{R}$</span>. Generally, linear regression is implemented based on normal equations, i.e., a method which gives you the best linear model that fits your data, rather than optimizing the model's weights by means of gradient-based optimization algorithms. Given a training set <span class="math-container">$X \in \mathbb{R}^{m \times n}$</span> and a target <span class="math-container">$y \in \mathbb{R}^{m}$</span>, the weights <span class="math-container">$\vartheta \in \mathbb{R}^{n}$</span> of the linear regression model are computed as follows:
  <span class="math-container">$$ \vartheta = \left(X^T X \right)^{-1} X^T y$$</span></p>
  <p>As you can see, an inverse matrix appears in the formula. In order to compute <span class="math-container">$A^{-1}$</span>, the constant <span class="math-container">$\frac{1}{|A|}$</span> must be computed, with <span class="math-container">$|A|$</span> being the determinant of <span class="math-container">$A$</span>. However, if <span class="math-container">$A$</span> has linear dependent rows/cols, <span class="math-container">$|A|=0$</span>, thus <span class="math-container">$\frac{1}{|A|}$</span> is not defined. This also implies that <span class="math-container">$A^{-1}$</span> cannot be computed, and you cannot train a linear regression model under these assumptions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T22:19:14.547Z
Answers.Text: Thanks for the accurate and thorough response!
Answers.UserDisplayName: Amirreza A.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T13:09:24.390Z
Answers.LastActivityDate: 2024-07-11T13:09:24.390Z
Answers.UserDisplayName: frad
Body: <p>I have a dataset that consists of data about students. The features are things such as <code>passed credits</code>, <code>failed credits</code>, <code>semester</code> among with other features. The target is the student's gpa. I believe that while the <code>passed credits</code> and <code>failed credits</code> don't have a direct correlation to the gpa (since they depend on the amount of the time students spent on the school), the values of <code>passed credits / semester</code> and <code>failed credits / semester</code> highly affect the outcome of the model. I want to know what is the best method of passing the data to a linear regression model? Should I pass <code>passed credits / semester</code> and <code>failed credits / semester</code> to the model and ignore <code>passed credits</code> and <code>failed credits</code> or vice versa? Or should I leave the data untouched and the model will figure out the relation itself?</p>
  
Tags: <linear-regression><features><feature-engineering>
Title: Is a linear regression model able to figure out the relation of division among two features?
UserDisplayName: Amirreza A.
ThreadText: # Is a linear regression model able to figure out the relation of division among two features?
  
  (Asked by: username_0 on 2024-05-29)
  
  I have a dataset that consists of data about students. The features are things such as `passed credits`, `failed credits`, `semester` among with other features. The target is the student's gpa. I believe that while the `passed credits` and `failed credits` don't have a direct correlation to the gpa (since they depend on the amount of the time students spent on the school), the values of `passed credits / semester` and `failed credits / semester` highly affect the outcome of the model. I want to know what is the best method of passing the data to a linear regression model? Should I pass `passed credits / semester` and `failed credits / semester` to the model and ignore `passed credits` and `failed credits` or vice versa? Or should I leave the data untouched and the model will figure out the relation itself?
  
  **Tags:** linear-regression, features, feature-engineering
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-11. Score: 1 | Accepted answer
  
  What you mentioned is called **feature engineering**, *i.e.*, the process of using domain knowledge to create new features or modify existing ones in a way that makes machine learning algorithms work more effectively.
  
  Given a training set $X=\left\{x^{(i)}, y^{(i)} \right\}\_{i=1}^{m}$, with $x^{(i)} \in \mathbb{R}^{n}$ being the $i$-th sample, and $y^{(i)} \in \mathbb{R}$ the outcome, if you think the ratio of two features $X\_1/X\_2$ is more relevant to the prediction of $y$ than $X\_1$ and $X\_2$ separately, then you should compute this new hand-crafted feature, and drop $X\_1$ and $X\_2$ (see avoiding collinearity below).
  
  However, training a linear regression on $X\_1$ and $X\_2$ versus training on the new feature $\frac{X\_1}{X\_2}$ results in fundamentally different models. The linear regression model is defined by:
  $$
  \hat{y}^{(i)} = \vartheta\_0 + \vartheta\_1 x^{(i)}\_1 + \vartheta\_2 x^{(i)}\_2 + \ldots + \vartheta\_n x^{(i)}\_n
  $$
  where $\hat{y}^{(i)}$ is the predicted value for the $i$-th sample. By contrast, if we introduce the new feature $\frac{x^{(i)}\_1}{x^{(i)}\_2}$, the model becomes:
  $$
  \hat{y}^{(i)} = \vartheta\_0 + \vartheta\_1 \frac{x^{(i)}\_1}{x^{(i)}\_2} + \vartheta\_3 x^{(i)}\_3 + \ldots + \vartheta\_n x^{(i)}\_n
  $$
  Here, the ratio $\frac{X\_1}{X\_2}$ is weighted by a single parameter, $\vartheta\_1$, instad of $\vartheta\_1$ and $\vartheta\_2$ separetely. This modification changes the model’s architecture (*i.e.*, the number of trainable parameters $\vartheta\_k$) and therefore the model's performance.
  
  ## Avoid collinearity
  
  A good rule is that of dropping from your training set linearly dependent features. Suppose you have two different features, $X\_1$ and $X\_2$, that are linearly dependent (or collinear):
  $$X\_2 = aX\_1 + b$$
  where $a, b \in \mathbb{R}$. Generally, linear regression is implemented based on normal equations, i.e., a method which gives you the best linear model that fits your data, rather than optimizing the model's weights by means of gradient-based optimization algorithms. Given a training set $X \in \mathbb{R}^{m \times n}$ and a target $y \in \mathbb{R}^{m}$, the weights $\vartheta \in \mathbb{R}^{n}$ of the linear regression model are computed as follows:
  $$ \vartheta = \left(X^T X \right)^{-1} X^T y$$
  
  As you can see, an inverse matrix appears in the formula. In order to compute $A^{-1}$, the constant $\frac{1}{|A|}$ must be computed, with $|A|$ being the determinant of $A$. However, if $A$ has linear dependent rows/cols, $|A|=0$, thus $\frac{1}{|A|}$ is not defined. This also implies that $A^{-1}$ cannot be computed, and you cannot train a linear regression model under these assumptions.
  
  > **Comments:**
  > **username_0**: Thanks for the accurate and thorough response!  
  
  ---
  
  