Sub-sampling a dataset to a different target distribution without replacement - bias correction? Suppose i have dataset $X$ and $Y$, and i want to sub-sample from $X$ so that the resulting (sample) distribution is as close to that of $Y$ as possible.
One thing i can do is subsample with probability (density) $\frac{d_y}{d_x}(X)$.
However, i notice that if i subsample without replacement some bias is introduced, whereas if i do so with replacement the resulting distribution is more similar to my target.
Question: Is there any way to sample from $X$ without replacement and still achieve a distribution that is similar to $Y$?
I have assumed the issue is sampling without replacement but if it is actually something else i am happy to be corrected.
Some code to illustrate the scenario:
set.seed(1235)

X=rexp(10000,rate=1)
Y=rexp(10000,rate=2)
mean(X);mean(Y)


## want to sub-sample from X to so that the resulting distribution is similar to that of Y
estimated_density=density(X,bw='SJ')
target_density=density(Y,bw='SJ')
d0=approx(estimated_density,xout=X)$y
d1=approx(target_density,xout=X)$y
d1=ifelse(is.na(d1),0,d1)


my_sub_sample=function(replace){
  summary(replicate(500,{
    subsample=sample(X,size = 5000,prob=d1/d0,replace = replace)
    mean(subsample)   
  }))

}

my_sub_sample(replace=T) ## mean close to 0.5
my_sub_sample(replace=F) ## mean not close to 0.5


For arbitrary distributions, there isn't any way to ensure that a subsample of $X$ has a similar distribution to $Y$, whether you use replacement or not. Consider the distributions where $X \sim U(1,2)$ and $Y \sim U(3,4)$ - it's not even possible to draw a single value that would occur in both distributions, much less draw a sample from $X$ that is representative of $Y$.
The issue you have in your example is that even though the distributions are overlapping, your subsample is very large, at half the total population of $X$. Without replacement, a very large subsample of $X$ can only deviate so far from $X$, but how that deviates from $Y$ will require a direct comparison of the the $X$ and $Y$ distributions. But if you could do that, it's not clear why you'd bother subsampling $X$ instead of just drawing from $Y$ in the first place.
Suppose you want to draw from $X \sim N(0,1)$ to approximate $Y \sim N(5,1)$, with a population of $N=1000$. Only  a handful of samples from $X$ have values that are "useful" for representing $Y$, with values close to 5. If you sample without replacement, once those few samples from $X$ are "used up", you only have values that are highly inconsistent with $Y$ remaining. But there's no straightforward correction pattern, you'd have the same issue with opposite directionality if you were trying to approximate $Y \sim N(-5,1)$. Any fix will involve going back to the original $Y$ distribution to find what parts are being under-represented by the sample from $X$.
