My guess is that under some conditions the GPU makes slight approximations that can differ from one run to another.
Try counting the gradients that differ by more than some small amount.
epsilon = 1e-8
print("params have same gradient?: ", 
    p1.data.numel() == ((p1.grad.data - p2.grad.data).abs() <= epsilon).nonzero().size(0))

Note that strictly speaking the models do not share parameters. They are just identically initialised. When one model is trained, the parameters of the other model will not change.