Testing the difference of proportions equal to a certain value Suppose there are two samples $X_1\dots X_n$, $Y_1\dots Y_m$ of binary variables $X\sim Bern(p_1)$ and $Y\sim Bern(p_2)$.
Is there any way to test the following hypothesis by the exact test?
$$H_0: p_1-p_2 = p; $$
$$H_1: p_1-p_2 \not= p,$$
where $p \not = 0$.
I do not want to use t-test since in my design $p_1$ and $p_2$ are small and I'm not sure about proper convergence of t-statistic.

I don't think there is an exact test in this case, but there is an approximate test. In general, concerns about the poor approximation of the approximate test are likely to be exaggerated unless you are exceedingly unfortunate to have both very very low $p_1$, $p_2$ and very very low $n$, $m$.
Certainly, higher $n$, $m$ will help, as the plot below shows. Note: I defined $p_2$=$p_1$+$p$.

Let x and y denote the numbers of successes observed in two independent sets of n and m Bernoulli trials, respectively, where $p_1$ and $p_2$ are the true success probabilities associated with each set of trials. Let $p_e=\frac{x+y}{n+m}$ and define:
$$z=\frac{\frac{x}{n}-\frac{y}{m}-(p_1-p_2)}{\sqrt{\frac{p_e(1-p_e)}{n}+\frac{p_e(1-p_e)}{m}}}$$
$z$ is approximately ~ $Normal(0,1)$.

In your particular case, with m and n around 150, the approximation is very good as long as the smallest of the 2 probabilities is no less than ~ 0.04. I colored the sampling distribution in blue when $p_1>=0.04$ and in red otherwise.

#code for the first plot
p1=0.05
p=0.03
p2=p1+p
nvalue=20
mvalue=25

zhats<-NULL
for (i in 1:10000) {
  set.seed(i)
  data1<-rbinom(n=nvalue,size=1,p=p1)
  set.seed(i+20)
  data2<-rbinom(n=mvalue,size=1,p=p2)
  p_e<-(sum(data1)+sum(data2))/(nvalue+mvalue)
  z_hat<-(mean(data1)-mean(data2)+p)/sqrt(p_e*(1-p_e)/nvalue+p_e*(1-p_e)/mvalue)
  zhats<-c(zhats,z_hat)
}

plot(density(zhats),col="red",xlab="",main=paste0("n=",nvalue," m=",mvalue," p1=",p1," p=",p),lwd=2,
     xlim=c(-4,4),ylim=c(0,0.5),cex.main=1.7)
par(new=T)
plot(density(rnorm(n=1000000)),xlim=c(-4,4),ylim=c(0,0.5),ann=F,ylab=F,lwd=2)

