Great, thank you! I will try replacing the activations as you suggested. In my case it looks like it will be more appropriate to get the untrained DenseNet and replace the activations and BNs and train on my dataset.