Sub-sampling a dataset to a different target distribution without replacement - bias correction? Suppose i have dataset $X$ and $Y$, and i want to sub-sample from $X$ so that the resulting (sample) distribution is as close to that of $Y$ as possible.
One thing i can do is subsample with probability (density) $\frac{d_y}{d_x}(X)$.
However, i notice that if i subsample without replacement some bias is introduced, whereas if i do so with replacement the resulting distribution is more similar to my target.
Question: Is there any way to sample from $X$ without replacement and still achieve a distribution that is similar to $Y$?
I have assumed the issue is sampling without replacement but if it is actually something else i am happy to be corrected.
Some code to illustrate the scenario:
set.seed(1235)

X=rexp(10000,rate=1)
Y=rexp(10000,rate=2)
mean(X);mean(Y)


## want to sub-sample from X to so that the resulting distribution is similar to that of Y
estimated_density=density(X,bw='SJ')
target_density=density(Y,bw='SJ')
d0=approx(estimated_density,xout=X)$y
d1=approx(target_density,xout=X)$y
d1=ifelse(is.na(d1),0,d1)


my_sub_sample=function(replace){
  summary(replicate(500,{
    subsample=sample(X,size = 5000,prob=d1/d0,replace = replace)
    mean(subsample)   
  }))

}

my_sub_sample(replace=T) ## mean close to 0.5
my_sub_sample(replace=F) ## mean not close to 0.5


It looks like you are trying to use importance sampling, which is the technique that I would have suggested for this problem.
First, a note: your implementation is different to the usual way of implementing importance sampling.  Typically we would sample from a proposal distribution $x_i \sim q(\cdot)$ and weight the resulting samples by importance weights calculated by $p(x_i) / q(x_i)$, where $p(\cdot)$ is the target distribution.  Instead of sampling directly from the proposal, you are sampling in proportion to the importance weights (similar to this post).  I believe that is fine and shouldn't cause problems.
Unfortunately, there's a mismatch between the proposal distribution you used to do the sampling (resampling from $X$ without replacement) and what you used in the denominator of the importance weights (a density estimate of $X$ used in an iid fashion, i.e. assuming sampling with replacement).  I suspect it is the with/without replacement mismatch between these two parts that is the main cause of the bias.
If you could sample with replacement, then you shouldn't have any issues (as illustrated by your code).  I presume this is not an option for your particular application, otherwise you would have just done it that way.
If you want to sample without replacement, then one idea might be to adapt the denominator accordingly.  It needs to correctly specify the distribution of a set of values that are sampled without replacement from $X$.
I'm not sure if there's an easy way to do this.  You could try doing it iteratively as you sample each observation in turn (in your code above, you would need to re-run density() after drawing each element of X), but that would likely be computationally prohibitive.  There might be a better way, where you do a calculation for a whole sample from X all at once, but I can't immediately think of how to implement this.
By the way, if you only ever draw a small sample relative to the total population, the bias should be quite small too.
