Answers.Body: <p>The problem you are portraying looks like a modified <a href="https://medium.com/@jayeshbahire/the-xor-problem-in-neural-networks-50006411840b" rel="nofollow noreferrer">XOR problem</a>. You can't throw away the lines with a label of 1 because a the model won't be able to learn this class.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-24T00:13:55.467Z
Answers.LastActivityDate: 2019-08-24T00:13:55.467Z
Answers.UserDisplayName: CaucM
Answers.Body: <p>This is perfectly acceptable in a stochastic environment. Generally your loss is to minimize <span class="math-container">$-log\ p(Y|X)$</span> or equivalently <span class="math-container">$-\sum_i log\ p(y_i|x_i)$</span>. This optimization is equivalent to <span class="math-container">$-\mathbb{E}\log\ p(y_i|x_i)$</span>. In other words you are minimizing in this case:  </p>
  
  <p><span class="math-container">$$ 
  \begin{align*}  
  L &amp;= -log\ p(1|x_0) - log\ p(0|x_0) \\ 
  &amp;= -log [p(1|x_0) * p(0|x_0)] \\
  &amp;= -log [p(1|x_0) * (1 - p(1|x_0))] \\
  \end{align*}
  $$</span><br>
  or since log is concave equivalently minimizing<br>
  <span class="math-container">$$ \hat L = -p(1|x_0) * (1 - p(1|x_0)) $$</span> 
  After some basic calc 1, we see the optimal result we want the system to learn is that<br>
  <span class="math-container">$$ p(1|x_0) = .5$$</span>  </p>
  
  <p>Note that if you had more evidence, the result would just be that you want it to learn that it is <span class="math-container">$1$</span> with probability <span class="math-container">$\mathbb{E}_i\ y_i | x$</span> </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-26T03:03:10.813Z
Answers.Text: So throwaway  or keep both columns?
Answers.UserDisplayName: NebulousReveal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-26T03:04:26.713Z
Answers.Text: keep both. Do **not** throw away data unless you have good reason to. In this case you want your model to output .5 (not 0 or 1)
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-26T03:09:05.523Z
Answers.Text: What happens if the outcome variable for both rows (or $n$ rows) are the same (i.e we have duplicate rows)? Should we throw one of them out? Or keep them both? Does it really matter?
Answers.UserDisplayName: NebulousReveal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-26T11:08:46.550Z
Answers.Text: @Prime In that case it depends, if this is due to a sampling scheme then do not, because usually thatll mean that daat point is twice as important on the other hand if someone accidentally copy and pasted a row, then yes delete it, because itll be giving additionall importance to a point that doesnt deserve it
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-24T15:37:00.547Z
Answers.LastActivityDate: 2019-08-24T15:37:00.547Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>I might consider 2 models (throw away col 1 and throw away col 4), and one more that keeps both, and see which generalises better to test set.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-04T01:25:30.100Z
Answers.LastActivityDate: 2019-09-04T01:25:30.100Z
Answers.UserDisplayName: joek47
Body: <p>Suppose we have a labeled data set with columns <span class="math-container">$A$</span>, <span class="math-container">$B$</span>, and <span class="math-container">$C$</span> and a binary outcome variable <span class="math-container">$X$</span>. Suppose we have rows as follows:</p>
  
  <pre><code> col  A B C X
    1   1 2 3 1
    2   4 2 3 0
    3   6 5 1 1
    4   1 2 3 0
  </code></pre>
  
  <p>Should we throw away either row 1 or row 4 because they have different values of the outcome variable X? Or keep both of them?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-30T12:37:03.860Z
Comments.Text: I think your question is quite naive. If you can share your motivation for the question, then the question would attract more apt answers.
Comments.UserDisplayName: naive
Tags: <machine-learning><ai-design><training><data-science>
Title: What should we do when we have equal observations with different labels?
UserDisplayName: guest_guy
ThreadText: # What should we do when we have equal observations with different labels?
  
  (Asked by: username_0 on 2019-08-23)
  
  Suppose we have a labeled data set with columns $A$, $B$, and $C$ and a binary outcome variable $X$. Suppose we have rows as follows:
   col  A B C X
    1   1 2 3 1
    2   4 2 3 0
    3   6 5 1 1
    4   1 2 3 0
  
  Should we throw away either row 1 or row 4 because they have different values of the outcome variable X? Or keep both of them?
  
  **Tags:** machine-learning, ai-design, training, data-science
  > **Comments:**
  > **username_1**: I think your question is quite naive. If you can share your motivation for the question, then the question would attract more apt answers.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2019-08-24. Score: 4
  
  The problem you are portraying looks like a modified [XOR problem](https://medium.com/@jayeshbahire/the-xor-problem-in-neural-networks-50006411840b). You can't throw away the lines with a label of 1 because a the model won't be able to learn this class.
  
  
  ---
  
  ### Answer by username_3 on 2019-08-24. Score: 1
  
  This is perfectly acceptable in a stochastic environment. Generally your loss is to minimize $-log\ p(Y|X)$ or equivalently $-\sum\_i log\ p(y\_i|x\_i)$. This optimization is equivalent to $-\mathbb{E}\log\ p(y\_i|x\_i)$. In other words you are minimizing in this case:
  
  $$
  \begin{align\*}
  L &= -log\ p(1|x\_0) - log\ p(0|x\_0) \\
  &= -log [p(1|x\_0) \* p(0|x\_0)] \\
  &= -log [p(1|x\_0) \* (1 - p(1|x\_0))] \\
  \end{align\*}
  $$  
  or since log is concave equivalently minimizing  
  $$ \hat L = -p(1|x\_0) \* (1 - p(1|x\_0)) $$
  After some basic calc 1, we see the optimal result we want the system to learn is that  
  $$ p(1|x\_0) = .5$$
  
  Note that if you had more evidence, the result would just be that you want it to learn that it is $1$ with probability $\mathbb{E}\_i\ y\_i | x$
  
  > **Comments:**
  > **username_4**: So throwaway  or keep both columns?  
  > **username_3**: keep both. Do **not** throw away data unless you have good reason to. In this case you want your model to output .5 (not 0 or 1)  
  > **username_4**: What happens if the outcome variable for both rows (or $n$ rows) are the same (i.e we have duplicate rows)? Should we throw one of them out? Or keep them both? Does it really matter?  
  > **username_3**: @Prime In that case it depends, if this is due to a sampling scheme then do not, because usually thatll mean that daat point is twice as important on the other hand if someone accidentally copy and pasted a row, then yes delete it, because itll be giving additionall importance to a point that doesnt deserve it  
  
  ---
  
  ### Answer by username_5 on 2019-09-04. Score: 0
  
  I might consider 2 models (throw away col 1 and throw away col 4), and one more that keeps both, and see which generalises better to test set.
  
  
  ---
  
  