How to detect and remove the duplicate samples Suppose $X = \{(x_1,y_1),\cdots,(x_N,y_N)\}$ is our samples. Now if I currently copy paste $X$ one time, then our sample data will become $(X,X).$ And we know that the half of samples, the copy pasted part is meanless. So in pre-process of data, how to detect and deal with such samples?
Since we know that such samples will have the impact. For example linear regression, copy pasting the sample will not affect the coefficient of estimation and R-squared, however it will affect the estimation variance and T test.

To detect (near-)duplicates, you can compute nearest-neighbour distances.  If you have a lot of variables this could be slow; one speed-up is to do it for a fixed or random subsample of variables first.
(A speed-up with better theoretical guarantees is to take a random low-dimensional projection and compute nearest-neighbours for the projected variables, but that's more complicated)
