Answers.Body: <p>This depends on:</p>
  <ol>
  <li>Whether the timestamp is additional information. I.e. is the temporal dimension relevant?</li>
  <li>Removing samples will shift the distribution of the data set. I.e. if you have 2 possible states, with 90 copies of 1 and 10 copies of the other, removing all duplicates means the model will not come into contact with the 90/10 ratio in the data, but will see a 1/1 ratio, this can bias your model towards the underrepresented class, reducing performance.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T07:04:00.693Z
Answers.Text: Duplicate records will leak into test set. Isn't this important? Is there any workaround?
Answers.UserDisplayName: Free Palestine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T07:55:11.083Z
Answers.Text: You're worried you'll have identical samples in both train and test? What are you using a test set for in the context of a DQN?
Answers.UserDisplayName: Kroshtan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T08:01:36.940Z
Answers.Text: Yes, I'm using sensors and a actuator statuses as train set inorder to learn what action to take for the actuator, then I want to test the algorithm with test set to see the accuracy. and timestamp is not important
Answers.UserDisplayName: Free Palestine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T08:11:51.683Z
Answers.Text: You're training a classifier, disguised as an RL algorithm. A DQN algorithm trains on reward, not labels, and so the concept of accuracy makes little sense. Also, because a DQN network learns sequentially, multiple identical sample are still relevant, because after each update, the output of the model is different, and thus the update is, too. For classification, the duplicated data is very much a problem. For reference: [On using test data in RL](https://ai.stackexchange.com/questions/32497/do-we-use-validation-and-test-sets-for-training-a-reinforcement-learning-agent)
Answers.UserDisplayName: Kroshtan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-08T14:59:21.427Z
Answers.LastActivityDate: 2022-08-08T14:59:21.427Z
Answers.UserDisplayName: Kroshtan
Body: <p>I have a a dataset that most of records and their corresponding labels are the same and only timestamp of each records is different from other record.
  If I ignore duplicate records in for training some algorithm like DQN, is this a correct approach?</p>
  
Tags: <machine-learning><training>
Title: Can I ignore duplicate records of dataset for training?
UserDisplayName: Free Palestine
ThreadText: # Can I ignore duplicate records of dataset for training?
  
  (Asked by: username_0 on 2022-08-06)
  
  I have a a dataset that most of records and their corresponding labels are the same and only timestamp of each records is different from other record.
  If I ignore duplicate records in for training some algorithm like DQN, is this a correct approach?
  
  **Tags:** machine-learning, training
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-08. Score: 1
  
  This depends on:
  
  1. Whether the timestamp is additional information. I.e. is the temporal dimension relevant?
  2. Removing samples will shift the distribution of the data set. I.e. if you have 2 possible states, with 90 copies of 1 and 10 copies of the other, removing all duplicates means the model will not come into contact with the 90/10 ratio in the data, but will see a 1/1 ratio, this can bias your model towards the underrepresented class, reducing performance.
  
  > **Comments:**
  > **username_0**: Duplicate records will leak into test set. Isn't this important? Is there any workaround?  
  > **username_1**: You're worried you'll have identical samples in both train and test? What are you using a test set for in the context of a DQN?  
  > **username_0**: Yes, I'm using sensors and a actuator statuses as train set inorder to learn what action to take for the actuator, then I want to test the algorithm with test set to see the accuracy. and timestamp is not important  
  > **username_1**: You're training a classifier, disguised as an RL algorithm. A DQN algorithm trains on reward, not labels, and so the concept of accuracy makes little sense. Also, because a DQN network learns sequentially, multiple identical sample are still relevant, because after each update, the output of the model is different, and thus the update is, too. For classification, the duplicated data is very much a problem. For reference: [On using test data in RL](https://ai.stackexchange.com/questions/32497/do-we-use-validation-and-test-sets-for-training-a-reinforcement-learning-agent)  
  
  ---
  
  