It’s the same in both cases - the downsizing is not at the top or bottom of the main “trunk” path, but rather in the parallel shortcut path.
In the caffe case it’s the res5a_branch1 convolution that’s doing the downsizing. In the PyTorch case note that it’s downsample(x), not downsample(out) - this is the parallel shortcut branch, not an operation  being added after bn3(out).
Ben