Answers.Body: <p>Ideal advise is to feed the raw data to the neural networks to let neural networks make its own inference</p>
  <p>Considering you have expert knowledge that <span class="math-container">$X1/X2$</span> has effect on <span class="math-container">$Y$</span> , here the new feature (<span class="math-container">$X1/X2$</span>) is referred to as a derived feature</p>
  <p>However, there are few advantages which can help you consider of using derived features like <span class="math-container">$X1/X2$</span> in your case</p>
  <ul>
  <li>Being a domain expert or SME choosing X1/X2 as an important feature, you can ideally accelerate the training process</li>
  <li>Highly advantageous when you are short on CPU time</li>
  <li>Most neural networks calculate sums fed through the activation functions, estimating the ratio or the multiplication requires lot of neurons especially if <span class="math-container">$X1/X2$</span> is an important feature</li>
  <li>You can use feature pruning techniques to eliminate redundant features</li>
  <li>One major drawback of neural network is often considered to be a black box model that’s the approximation of neural networks doesn’t give any insight of the form of function f. Use make use of feature selection algorithms to pare down feature space.</li>
  <li>With large number of parameters in model also increasing the risk of overfitting the network. You can however overcome this with good regularisation methods. You can also use feature pruning to avoid overfitting when you are having limited data.</li>
  </ul>
  <p>I found a literature in the field of medicine that deals with deriving features based on expertise accurately identified the required states.</p>
  <p><a href="https://bmcanesthesiol.biomedcentral.com/articles/10.1186/s12871-021-01285-x" rel="nofollow noreferrer">Heart rate variability-derived features based on deep neural network for distinguishing different anaesthesia states</a></p>
  <blockquote>
  <p>The incorporation of four HRV-derived features in the time and frequency domain and a deep neural network could accurately distinguish between different anaesthesia states.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T19:27:32.813Z
Answers.LastActivityDate: 2021-11-23T06:45:21.180Z
Answers.LastEditDate: 2021-11-23T06:45:21.180Z
Answers.UserDisplayName: Archana David
Answers.Body: <p>Model and Objective function are playing together. If you can have an objective function that somehow exclude this relation that you have in your mind, then the model can focus on learning to predict based on other information. Then you have trained the model, but if your downstream task should consider that at the end, you could manually add this relation and apply the relation you have in your mind at the end. That's my philosophical answer, but the question is how do you want to implement it?‌‌‌‌‌‌‌‌‌‌ It depends on details of the project, and this approach might not be feasible, but I guess your concern is real. The model is inclined to learn the shortest path to the answer, and if you provide it for it, then it will use it.
  If I were you, I might try to first normalize the features so they'd have similar effects. I don't know how this would be implemented in your problem though.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-11T06:22:21.957Z
Answers.LastActivityDate: 2021-11-11T06:22:21.957Z
Answers.UserDisplayName: Yousef
Answers.Body: <p>You are refering to the first and very important step in a machine learning process called <em>data preprocessing</em>. Refering to <a href="https://towardsdatascience.com/data-preprocessing-concepts-fa946d11c825" rel="nofollow noreferrer">this article</a>, inside data preprocessing there are many smaller processes that deal with features: <em>feature extraction</em>, <em>feature selection</em>, <em>feature aggregration</em> and <em>feature encoding</em> to name a few.</p>
  <p>The idea of creating new features out of raw features is not new, but rather a well known concept in machine learning called <a href="https://www.mathworks.com/discovery/feature-extraction.html" rel="nofollow noreferrer">feature extraction</a>. Suppose you decide to create the new <span class="math-container">$X_1 / X_2$</span> feature and <strong>add</strong> it to the raw set of features <span class="math-container">${X_1, X_2}$</span>, because you are completely sure that it has an effect on <span class="math-container">$Y$</span>, then that is a perfectly reasonable example of feature extraction.</p>
  <p>Refering to the article above on <a href="https://www.mathworks.com/discovery/feature-extraction.html" rel="nofollow noreferrer">feature extraction</a>, notice that nowadays there is some growing consensus that when using deep neural networks, the first hidden layers of the network can serve the purpose of <em>automatic</em> feature extraction, without you manually adding the new features to the raw set of features.  This premise stems from the fact that deep neural networks are essentially non-linear function approximators, so the first hidden layers can approximate any feature extraction that you do manually.</p>
  <p>You can understand that in order to do automatic feature extraction, you might need more hidden layers and more neurons in each layer, to increase the computing power of your model. There is also the downside that you <em>can not know for sure</em> what is happening in the first hidden layers, if it is doing feature extraction the way you intended or not.</p>
  <p>In your case, my advice would be to <em>do</em> the manual addition of <span class="math-container">$X_1 / X_2$</span> in the set of features, if you are only adding one such fraction or <em>a handful</em>. Also, if you are completely sure there is a correlation between those fractions and <span class="math-container">$Y$</span>. If on the other hand, you are adding more than a handful of such fractions, then it <em>does not</em> make sense to add them all. It is hard to be completely sure that all the fractions have an effect on <span class="math-container">$Y$</span>. It would also exhaust the memory space, by having a lot of features on the input layer. In that case I would suggest that you skip manual feature extraction, increase the number of hidden layers and let the network do its auto-magic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-16T09:37:47.693Z
Answers.Text: Hi, I wonder if he couldn't just replace X1 and X2 with the X1/X2 fraction. I often do such things when I have good understanding of data. I can calculate correlated between input features and output and see that fraction is more correlated than separated features itself.  There are many tools to do that in libraries like scipy or numpy.
Answers.UserDisplayName: MASTER OF CODE
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T14:46:30.140Z
Answers.Text: Yes, it is perfectly reasonable: it is called feature aggregation.
Answers.UserDisplayName: devidduma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T16:34:21.290Z
Answers.LastActivityDate: 2021-11-15T18:15:51.713Z
Answers.LastEditDate: 2021-11-15T18:15:51.713Z
Answers.UserDisplayName: devidduma
Body: <p><strong>This question is assuming a sequential, deep neural network</strong></p>
  <p>Given some features [X1, X2, ... Xn], I'm trying to predict some value Y.</p>
  <p>The raw data available to me contains feature <code>X1</code> and feature <code>X2</code>. Say that I <strong>know</strong> there is an effect on <code>Y</code> based on the <strong>ratio</strong> of the two features, i.e. <code>X1 / X2</code>.</p>
  <p><strong>Should I add a new feature, mathematically defined as the ratio of the two features?</strong> I haven't been able to locate any literature which begins to describe the necessity or warnings of this.</p>
  <p>Instinctly I'm worried about the following:</p>
  <ul>
  <li>Overfitting and the need for excessive regularization, due to duplicate information in the feature set</li>
  <li>Exponentially growing number of features, since defining a ratio between each feature <strong>may</strong> be necessary</li>
  </ul>
  <p>However, I also recognize that certain relationships are impossible to be defined by a deep neural network (i.e. logic gates, exponential relationships, etc), so when would this sort of &quot;relationship defining&quot; be necessary? For example, if an exponential relationship is known to exist?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T03:53:19.860Z
Comments.Text: I can't give you a mathematically founded answer, but as with a lot of machine learning, your best bet is probably to just test it. Do a run with and without the added features, and see how the network performs. If you can squeeze extra performance out of one or the other, then you'll likely have your answer
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T09:55:44.230Z
Comments.Text: just for clarification: when you say features [X1, X2, ... Xn] do you mean a set of features all presented at once to the model, i.e. a single training instance with n features, or since you stressed out that the model is sequential do you mean n training instances sorted by time or some other criteria?
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T18:22:35.197Z
Comments.Text: @EdoardoGuerriero I mean your first example; a single training instance with n features. When I say "sequential", I'm referring to sequential as in the network flows "from left to right".
Comments.UserDisplayName: Tyler M
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-11T15:36:13.570Z
Comments.Text: I totally agree with Recessive. I would just test it and see whether it is beneficial or not to have this new feature. As with many machine learning models, it is quite difficult to infer the effect of new features before really testing it.
Comments.UserDisplayName: PeterBe
Tags: <neural-networks><deep-learning>
Title: Should I allow NN to infer relationships of inputs?
UserDisplayName: Tyler M
ThreadText: # Should I allow NN to infer relationships of inputs?
  
  (Asked by: username_0 on 2021-10-18)
  
  **This question is assuming a sequential, deep neural network**
  
  Given some features [X1, X2, ... Xn], I'm trying to predict some value Y.
  
  The raw data available to me contains feature `X1` and feature `X2`. Say that I **know** there is an effect on `Y` based on the **ratio** of the two features, i.e. `X1 / X2`.
  
  **Should I add a new feature, mathematically defined as the ratio of the two features?** I haven't been able to locate any literature which begins to describe the necessity or warnings of this.
  
  Instinctly I'm worried about the following:
  
  * Overfitting and the need for excessive regularization, due to duplicate information in the feature set
  * Exponentially growing number of features, since defining a ratio between each feature **may** be necessary
  
  However, I also recognize that certain relationships are impossible to be defined by a deep neural network (i.e. logic gates, exponential relationships, etc), so when would this sort of "relationship defining" be necessary? For example, if an exponential relationship is known to exist?
  
  **Tags:** neural-networks, deep-learning
  > **Comments:**
  > **username_1**: I can't give you a mathematically founded answer, but as with a lot of machine learning, your best bet is probably to just test it. Do a run with and without the added features, and see how the network performs. If you can squeeze extra performance out of one or the other, then you'll likely have your answer  
  > **username_2**: just for clarification: when you say features [X1, X2, ... Xn] do you mean a set of features all presented at once to the model, i.e. a single training instance with n features, or since you stressed out that the model is sequential do you mean n training instances sorted by time or some other criteria?  
  > **username_0**: @EdoardoGuerriero I mean your first example; a single training instance with n features. When I say "sequential", I'm referring to sequential as in the network flows "from left to right".  
  > **username_3**: I totally agree with Recessive. I would just test it and see whether it is beneficial or not to have this new feature. As with many machine learning models, it is quite difficult to infer the effect of new features before really testing it.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_6 on 2021-11-15. Score: 3 | Accepted answer
  
  You are refering to the first and very important step in a machine learning process called *data preprocessing*. Refering to [this article](https://towardsdatascience.com/data-preprocessing-concepts-fa946d11c825), inside data preprocessing there are many smaller processes that deal with features: *feature extraction*, *feature selection*, *feature aggregration* and *feature encoding* to name a few.
  
  The idea of creating new features out of raw features is not new, but rather a well known concept in machine learning called [feature extraction](https://www.mathworks.com/discovery/feature-extraction.html). Suppose you decide to create the new $X\_1 / X\_2$ feature and **add** it to the raw set of features ${X\_1, X\_2}$, because you are completely sure that it has an effect on $Y$, then that is a perfectly reasonable example of feature extraction.
  
  Refering to the article above on [feature extraction](https://www.mathworks.com/discovery/feature-extraction.html), notice that nowadays there is some growing consensus that when using deep neural networks, the first hidden layers of the network can serve the purpose of *automatic* feature extraction, without you manually adding the new features to the raw set of features. This premise stems from the fact that deep neural networks are essentially non-linear function approximators, so the first hidden layers can approximate any feature extraction that you do manually.
  
  You can understand that in order to do automatic feature extraction, you might need more hidden layers and more neurons in each layer, to increase the computing power of your model. There is also the downside that you *can not know for sure* what is happening in the first hidden layers, if it is doing feature extraction the way you intended or not.
  
  In your case, my advice would be to *do* the manual addition of $X\_1 / X\_2$ in the set of features, if you are only adding one such fraction or *a handful*. Also, if you are completely sure there is a correlation between those fractions and $Y$. If on the other hand, you are adding more than a handful of such fractions, then it *does not* make sense to add them all. It is hard to be completely sure that all the fractions have an effect on $Y$. It would also exhaust the memory space, by having a lot of features on the input layer. In that case I would suggest that you skip manual feature extraction, increase the number of hidden layers and let the network do its auto-magic.
  
  > **Comments:**
  > **username_7**: Hi, I wonder if he couldn't just replace X1 and X2 with the X1/X2 fraction. I often do such things when I have good understanding of data. I can calculate correlated between input features and output and see that fraction is more correlated than separated features itself.  There are many tools to do that in libraries like scipy or numpy.  
  > **username_6**: Yes, it is perfectly reasonable: it is called feature aggregation.  
  
  ---
  
  ### Answer by username_4 on 2021-11-15. Score: 1
  
  Ideal advise is to feed the raw data to the neural networks to let neural networks make its own inference
  
  Considering you have expert knowledge that $X1/X2$ has effect on $Y$ , here the new feature ($X1/X2$) is referred to as a derived feature
  
  However, there are few advantages which can help you consider of using derived features like $X1/X2$ in your case
  
  * Being a domain expert or SME choosing X1/X2 as an important feature, you can ideally accelerate the training process
  * Highly advantageous when you are short on CPU time
  * Most neural networks calculate sums fed through the activation functions, estimating the ratio or the multiplication requires lot of neurons especially if $X1/X2$ is an important feature
  * You can use feature pruning techniques to eliminate redundant features
  * One major drawback of neural network is often considered to be a black box model that’s the approximation of neural networks doesn’t give any insight of the form of function f. Use make use of feature selection algorithms to pare down feature space.
  * With large number of parameters in model also increasing the risk of overfitting the network. You can however overcome this with good regularisation methods. You can also use feature pruning to avoid overfitting when you are having limited data.
  
  I found a literature in the field of medicine that deals with deriving features based on expertise accurately identified the required states.
  
  [Heart rate variability-derived features based on deep neural network for distinguishing different anaesthesia states](https://bmcanesthesiol.biomedcentral.com/articles/10.1186/s12871-021-01285-x)
  
  > The incorporation of four HRV-derived features in the time and frequency domain and a deep neural network could accurately distinguish between different anaesthesia states.
  
  
  ---
  
  ### Answer by username_5 on 2021-11-11. Score: 0
  
  Model and Objective function are playing together. If you can have an objective function that somehow exclude this relation that you have in your mind, then the model can focus on learning to predict based on other information. Then you have trained the model, but if your downstream task should consider that at the end, you could manually add this relation and apply the relation you have in your mind at the end. That's my philosophical answer, but the question is how do you want to implement it?‌‌‌‌‌‌‌‌‌‌ It depends on details of the project, and this approach might not be feasible, but I guess your concern is real. The model is inclined to learn the shortest path to the answer, and if you provide it for it, then it will use it.
  If I were you, I might try to first normalize the features so they'd have similar effects. I don't know how this would be implemented in your problem though.
  
  
  ---
  
  