Answers.Body: <p>You are doing a technique called &quot;feature engineering&quot;, where we manually add new features to the dataset that tell something about the data, so that the model is able to learn better. Every changes you make to the input data is already called Feature Engineering, even something small as adding a new number in the input data.</p>
  <p><a href="https://www.kaggle.com/learn/feature-engineering" rel="nofollow noreferrer">Kaggle</a> even has a dedicated course for Feature Engineering. Also, it has been empirically proven to help the model, such as in <a href="https://www.kaggle.com/code/prashant111/a-reference-guide-to-feature-engineering-methods/notebook" rel="nofollow noreferrer">here</a>, <a href="https://www.kaggle.com/code/willkoehrsen/introduction-to-manual-feature-engineering" rel="nofollow noreferrer">here</a>, and <a href="https://www.kaggle.com/code/seneralkan/advanced-feature-engineering" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-08T11:26:35.910Z
Answers.Text: but in my case the data will work when I adding number of sequence each array
Answers.UserDisplayName: stack offer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-08T11:27:12.730Z
Answers.Text: the ilustration as in this picture https://i.sstatic.net/Hd891.png
Answers.UserDisplayName: stack offer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-08T11:14:21.903Z
Answers.LastActivityDate: 2023-03-08T11:14:21.903Z
Answers.UserDisplayName: Minh-Long Luu
Body: <p>I am developing a regression model to analyze walking styles. The dataset I am using to build the model is from 2 different sources, let's call them dataset A and dataset B.</p>
  <p>Dataset A has a shape of <code>(15000,6)</code> and dataset B has a shape of <code>(15000,89)</code>. Both datasets are time-series data taken simultaneously for 5 minutes. I then trained the data using input data of shape <code>(15000,89,1)</code>. The <code>loss</code> and <code>val_loss</code> generated from the training process were very small, but when I tested it using real data, the model was unable to predict the real data. Therefore, I think I am experiencing overfitting.</p>
  <p>Then I tried adding new data to the input data. This new data is a sequence number from 1-50. I added this sequence number as an identity to the input data because the data I will generate will have 50 data in 1 second. I only added this sequence number to the input data. So the input data that was originally <code>(15000,89,1)</code> became <code>(15000,90,1)</code>. Then after adding this sequence number, I trained the new input data, and the result was better than before, and the model was able to predict real data. My question is, why is the technique of adding a sequence number to the input data able to avoid overfitting? Is the additional information of the sequence number part of data augmentation? If yes, please give me logical reasons and relevant references because I can't find relevant references and it is hard to find keywords for them.</p>
  <p>you can refer to this question to understand the question in detail: <a href="https://datascience.stackexchange.com/questions/117711/adding-data-sequences-as-unique-data-on-dataset-for-regression-model?noredirect=1#comment118506_117711">https://datascience.stackexchange.com/questions/117711/adding-data-sequences-as-unique-data-on-dataset-for-regression-model?noredirect=1#comment118506_117711</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-08T11:03:59.023Z
Comments.Text: First thing is I believe your problem is "underfitting". Second is, can you describe in details how you concat A and B to have shape `(15000, 89, 1)`?
Comments.UserDisplayName: Minh-Long Luu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-08T11:05:29.020Z
Comments.Text: no, I not concat the data. so the input data is (15000,89,1)
Comments.UserDisplayName: stack offer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-08T11:06:58.867Z
Comments.Text: in other hand I want predict dataset A using dataset B,so from (15000,89) will predict (15000,6)
Comments.UserDisplayName: stack offer
Tags: <convolutional-neural-networks><regression><data-augmentation>
Title: why by adding additional information as number of sequence on dataset can avoid overfitting
UserDisplayName: stack offer
ThreadText: # why by adding additional information as number of sequence on dataset can avoid overfitting
  
  (Asked by: username_0 on 2023-03-08)
  
  I am developing a regression model to analyze walking styles. The dataset I am using to build the model is from 2 different sources, let's call them dataset A and dataset B.
  
  Dataset A has a shape of `(15000,6)` and dataset B has a shape of `(15000,89)`. Both datasets are time-series data taken simultaneously for 5 minutes. I then trained the data using input data of shape `(15000,89,1)`. The `loss` and `val_loss` generated from the training process were very small, but when I tested it using real data, the model was unable to predict the real data. Therefore, I think I am experiencing overfitting.
  
  Then I tried adding new data to the input data. This new data is a sequence number from 1-50. I added this sequence number as an identity to the input data because the data I will generate will have 50 data in 1 second. I only added this sequence number to the input data. So the input data that was originally `(15000,89,1)` became `(15000,90,1)`. Then after adding this sequence number, I trained the new input data, and the result was better than before, and the model was able to predict real data. My question is, why is the technique of adding a sequence number to the input data able to avoid overfitting? Is the additional information of the sequence number part of data augmentation? If yes, please give me logical reasons and relevant references because I can't find relevant references and it is hard to find keywords for them.
  
  you can refer to this question to understand the question in detail: <https://datascience.stackexchange.com/questions/117711/adding-data-sequences-as-unique-data-on-dataset-for-regression-model?noredirect=1#comment118506_117711>
  
  **Tags:** convolutional-neural-networks, regression, data-augmentation
  
  > **Comments:**
  > **username_1**: First thing is I believe your problem is "underfitting". Second is, can you describe in details how you concat A and B to have shape `(15000, 89, 1)`?  
  > **username_0**: no, I not concat the data. so the input data is (15000,89,1)  
  > **username_0**: in other hand I want predict dataset A using dataset B,so from (15000,89) will predict (15000,6)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-08. Score: 0
  
  You are doing a technique called "feature engineering", where we manually add new features to the dataset that tell something about the data, so that the model is able to learn better. Every changes you make to the input data is already called Feature Engineering, even something small as adding a new number in the input data.
  
  [Kaggle](https://www.kaggle.com/learn/feature-engineering) even has a dedicated course for Feature Engineering. Also, it has been empirically proven to help the model, such as in [here](https://www.kaggle.com/code/prashant111/a-reference-guide-to-feature-engineering-methods/notebook), [here](https://www.kaggle.com/code/willkoehrsen/introduction-to-manual-feature-engineering), and [here](https://www.kaggle.com/code/seneralkan/advanced-feature-engineering).
  
  > **Comments:**
  > **username_0**: but in my case the data will work when I adding number of sequence each array  
  > **username_0**: the ilustration as in this picture https://i.sstatic.net/Hd891.png  
  
  ---
  
  