You could disable the running stats completely by setting track_running_stats=False.
However, your use case seems to be:

for x1 use batch stats and update running estimates
for x2 just use batch stats

What about the affine parameters?
Should they also be updated using the loss from x2 or just x1?
In the latter case, you could use two different batchnorm layers, pass a flag to forward, and switch between these layers depending if x1 or x2 was passed.