Answers.Body: <p>If you have the full dataset (comprising both &quot;ideal&quot;/cleaned and &quot;noisy&quot;/real examples), then no, it is not necessarily better to pretrain it with clean data just to initialise the weights and then update it with real examples. I don't see why it would be better to make the model overfit on clean data first and then correct it by using more noisy data, instead of just training it with all the data from the beginning.</p>
  <p>On the other hand, if you  have <em>only</em> clean data to start with (for example let us say you have some synthetic or human annotated data) then it could be worthwhile to pretrain the model with that to initialise the weights and when you get the real data update the model (so it could benefit the head start).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-17T04:42:15.077Z
Answers.Text: How about I have different sets? First one is ideal how the data is with timesteps 512, while the second one is more complex that the timestep is 3600. Specifically the data is an ECG signal where I'm using LSTM on it. In the medical domain, the 512 is a beat which represent foundation of entire signal. While 3600 timesteps is the real world scenerio, though the timesteps is actually has dynamic size,
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-17T04:57:11.080Z
Answers.Text: I mean, we can't train the data with different shape or different dimension?
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-17T07:16:46.157Z
Answers.Text: I don't think it will help you to train first on the "easy" 512 timesteps examples and then train on the "real" longer dynamic sized examples (the first para of my answer applies here). And dynamic length of the sequence in sequence models like LSTMs isn't considered a chance in shape or dimension. Sequence models are meant to handle this. Although that is probably a separate topic since this question is about the noise in the examples.
Answers.UserDisplayName: Sachin Hosmani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-16T15:16:18.580Z
Answers.LastActivityDate: 2024-08-16T15:16:18.580Z
Answers.UserDisplayName: Sachin Hosmani
Body: <p>Speaking generally, AFAIK Neural network relies on initial state at their weights. Mostly it started with random weights.</p>
  <p>Does that mean it will be better to train neural network with ideal data first? Then after the model's loss convergence, it will resume training with non-ideal data such as noisy data or something that related with real-world scenerio.</p>
  
Tags: <neural-networks><deep-learning><weights><transfer-learning><weights-initialization>
Title: Is it better to train neural network with ideal data before resuming with non-ideal data?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Is it better to train neural network with ideal data before resuming with non-ideal data?
  
  (Asked by: username_0 on 2024-08-16)
  
  Speaking generally, AFAIK Neural network relies on initial state at their weights. Mostly it started with random weights.
  
  Does that mean it will be better to train neural network with ideal data first? Then after the model's loss convergence, it will resume training with non-ideal data such as noisy data or something that related with real-world scenerio.
  
  **Tags:** neural-networks, deep-learning, weights, transfer-learning, weights-initialization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-16. Score: 2
  
  If you have the full dataset (comprising both "ideal"/cleaned and "noisy"/real examples), then no, it is not necessarily better to pretrain it with clean data just to initialise the weights and then update it with real examples. I don't see why it would be better to make the model overfit on clean data first and then correct it by using more noisy data, instead of just training it with all the data from the beginning.
  
  On the other hand, if you have *only* clean data to start with (for example let us say you have some synthetic or human annotated data) then it could be worthwhile to pretrain the model with that to initialise the weights and when you get the real data update the model (so it could benefit the head start).
  
  > **Comments:**
  > **username_0**: How about I have different sets? First one is ideal how the data is with timesteps 512, while the second one is more complex that the timestep is 3600. Specifically the data is an ECG signal where I'm using LSTM on it. In the medical domain, the 512 is a beat which represent foundation of entire signal. While 3600 timesteps is the real world scenerio, though the timesteps is actually has dynamic size,  
  > **username_0**: I mean, we can't train the data with different shape or different dimension?  
  > **username_1**: I don't think it will help you to train first on the "easy" 512 timesteps examples and then train on the "real" longer dynamic sized examples (the first para of my answer applies here). And dynamic length of the sequence in sequence models like LSTMs isn't considered a chance in shape or dimension. Sequence models are meant to handle this. Although that is probably a separate topic since this question is about the noise in the examples.  
  
  ---
  
  