As a side note, can you make the bias of conv layers before Batchnorm False. As batchnorm needs to standardize the layer outputs, having a bias increases a parameter and makes it difficult for the network to learn.