Answers.Body: <p>If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information.</p>
  <p>When it comes to correlation between data samples during training, this correlation is commonly broken up by training a network on randomly selected mini-batches of training data samples. So, you randomly sample e.g. 16 or 32 (or so) training examples based on which you apply a single update of the weights using some Stochastic Gradient Descent variant. Since the members of a mini-batch are sampled at random, chances for finding highly correlated training samples in some mini-batch shall be sufficiently minimized in order not to negatively affect the training outcome.</p>
  <p>Having said that, if you are concerned about overfitting of your model or weights that would overly weight just a small subset of all available input features, you could try applying regularization techniques like L1 (encouraging sparse representations) or L2 (encouraging low weights in general) regularization or dropout.
  In your particular case, since the main concern is an excessive contribution of only a small set of input features, L2 shall yield better results (avoiding excessively large weights that would be required to excessively much weight just a small number of features).</p>
  <p>Besides that: Commonly, you split your training dataset into 3 parts:</p>
  <ol>
  <li>Data used for fitting the model (actual training data)</li>
  <li>Data used for assessing the training progress &amp; possibly for determining when to apply early stopping (validation data)</li>
  <li>Test data used to assess the performance of the system after all training &amp; intermediate testing is done</li>
  </ol>
  <p>The final evaluation on the test dataset shall reveal then the generalization ability of your trained model to novel data.</p>
  <p>So, with regularization in place during training and relatively low error rates on the validation and test datasets, you are pretty much save even without checking for correlated data beforehand. Only when you really struggle decreasing the validation loss, it might be worth to further inspect what exactly is going wrong in terms of correlations and such.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T14:28:34.457Z
Answers.Text: Hey Daniel, thanks for the great explanation. Greatly appreciated!
Answers.UserDisplayName: Shogun187
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-13T09:25:34.530Z
Answers.LastActivityDate: 2020-07-13T09:56:56.137Z
Answers.LastEditDate: 2020-07-13T09:56:56.137Z
Answers.UserDisplayName: Daniel B.
Body: <p>I am new to working with neural networks. However, I have built some linear regression models in the past. My question is, is it worth looking for features with a correlation to my target variable as I would normally do in a linear regression or is it better to feed the neural network with all the data I have?</p>
  <p>Assuming that the data I have is all related to my target variable of course. I am working with this dataset and building a neural network regressor for it.</p>
  <p><a href="https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0101EN/labs/data/concrete_data.csv" rel="nofollow noreferrer">https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0101EN/labs/data/concrete_data.csv</a></p>
  <p>Here is a snippet of the data. The target variable is the concrete strength rate given a certain combination of materials for that concrete sample.</p>
  <p><a href="https://i.sstatic.net/0xaPT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0xaPT.png" alt="enter image description here" /></a></p>
  <p>I greatly appreciate any tips and explanations. I excuse me if this is too noob of a question but unfortunately I did not find any info about it on google. Thanks again!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-13T05:23:03.767Z
Comments.Text: If there is a correlation between some of the data, that means a couple of things. 1, you might be better of just doing linear regression or some simpler fitting as a NN might be overkill and 2, a neural network should learn very easily off this data. But it ultimately comes down to testing; if using all the data increases accuracy go with that, if not than just use a subset as it should make the model faster.
Comments.UserDisplayName: Recessive
Tags: <neural-networks><python><keras><linear-regression>
Title: Do correlations matter when building neural networks?
UserDisplayName: Shogun187
ThreadText: # Do correlations matter when building neural networks?
  
  (Asked by: username_0 on 2020-07-13)
  
  I am new to working with neural networks. However, I have built some linear regression models in the past. My question is, is it worth looking for features with a correlation to my target variable as I would normally do in a linear regression or is it better to feed the neural network with all the data I have?
  
  Assuming that the data I have is all related to my target variable of course. I am working with this dataset and building a neural network regressor for it.
  
  <https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0101EN/labs/data/concrete_data.csv>
  
  Here is a snippet of the data. The target variable is the concrete strength rate given a certain combination of materials for that concrete sample.
  
  [![enter image description here](https://i.sstatic.net/0xaPT.png)](https://i.sstatic.net/0xaPT.png)
  
  I greatly appreciate any tips and explanations. I excuse me if this is too noob of a question but unfortunately I did not find any info about it on google. Thanks again!
  
  **Tags:** neural-networks, python, keras, linear-regression
  
  > **Comments:**
  > **username_1**: If there is a correlation between some of the data, that means a couple of things. 1, you might be better of just doing linear regression or some simpler fitting as a NN might be overkill and 2, a neural network should learn very easily off this data. But it ultimately comes down to testing; if using all the data increases accuracy go with that, if not than just use a subset as it should make the model faster.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-13. Score: 1 | Accepted answer
  
  If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information.
  
  When it comes to correlation between data samples during training, this correlation is commonly broken up by training a network on randomly selected mini-batches of training data samples. So, you randomly sample e.g. 16 or 32 (or so) training examples based on which you apply a single update of the weights using some Stochastic Gradient Descent variant. Since the members of a mini-batch are sampled at random, chances for finding highly correlated training samples in some mini-batch shall be sufficiently minimized in order not to negatively affect the training outcome.
  
  Having said that, if you are concerned about overfitting of your model or weights that would overly weight just a small subset of all available input features, you could try applying regularization techniques like L1 (encouraging sparse representations) or L2 (encouraging low weights in general) regularization or dropout.
  In your particular case, since the main concern is an excessive contribution of only a small set of input features, L2 shall yield better results (avoiding excessively large weights that would be required to excessively much weight just a small number of features).
  
  Besides that: Commonly, you split your training dataset into 3 parts:
  
  1. Data used for fitting the model (actual training data)
  2. Data used for assessing the training progress & possibly for determining when to apply early stopping (validation data)
  3. Test data used to assess the performance of the system after all training & intermediate testing is done
  
  The final evaluation on the test dataset shall reveal then the generalization ability of your trained model to novel data.
  
  So, with regularization in place during training and relatively low error rates on the validation and test datasets, you are pretty much save even without checking for correlated data beforehand. Only when you really struggle decreasing the validation loss, it might be worth to further inspect what exactly is going wrong in terms of correlations and such.
  
  > **Comments:**
  > **username_0**: Hey username_2, thanks for the great explanation. Greatly appreciated!  
  
  ---
  
  