I’ll try adding batch norm and see what happens. I thought that I read in the ResNet paper
that shortcutting connections is a remedy to exploding/vanishing gradients, so I didn’t add batch norm. Maybe I mis-read. If it works out, I’ll report back.