 A mathematical illustration may illuminate the disagreement. For Poincaré, talk about what we might call ordinary geometry, the sense of space that we have prior to advanced instruction, is really about the ability we have to measure things. We can carry a rigid body around, and use it as a ruler. It is because we can do that that we can speak of the distance between places. If you want to make the set-up more abstract, there must be a space and a group that acts on the space and moves points in the space around. If this group has the property that however a region of that space is moved around it is never mapped onto a proper subset of itself then one can construct rigid bodies and talk about distance.