Dropout2d drops random channels from each image, their drop_connect drops whole images from the batch.
Therefore, every image is only seen by a subset of layers.
I am not sure what is the ‘correct’ way of doing DropConnect, but it seems to work reasonably well.