The torch.max operation in:
final_out =  torch.max(self.final(fc3_out), dim=1)[0]

will allow the gradient to pass to the max. value only and will set all other values to zero, so I’m not sure if that’s really what you want.
If you are dealing with e.g. a multi-class classification, pass the raw logits to nn.CrossEntropyLoss instead of using torch.max or applying softmax.