Hello,
It seems you gradient value is vanishing after you layer named self.conv4. Your structure is approaching like Encoder-Decoder model, because of that you can use a method which I recommend to you is that implement Residual Networks (a.k.a. ResNet ) to your structure for avoid vanishing gradient.
Besides decreasing learning rate will help you to avoid that problem.