BatchNorm and Dropout layers have different behavior in training and evaluation modes. Try to narrow down your problem by setting all the layers to “eval” mode except your BatchNorm layers. Then try the same thing with the Dropout layers.
BatchNorm uses a running average of mean and variance during training to compute the statistics used during eval mode. Check the running_mean and running_var properties on your BatchNorm layers.
One thing I’ve seen mess up BatchNorm is sending in uninitialized inputs in train mode. For example:
output_size = network(torch.FloatTensor(16, 3, 224, 224).cuda()).size() # BAD!

The uninitialized input can mess up the running averages, especially if it happens to contain NaNs.