Would it work to use the original images, feed them to the VAE, resize the output, and feed to the SSD?
It would be similar to this small example, which would still preserve the gradients in the smaller image, since the interpolation is differentiable:
x = torch.randn(3, 24, 24, requires_grad=True)
x_res = F.interpolate(x, size=30)

loss = x_res.mean()
loss.backward()
print(x.grad)