# layers
self.hidden_layer_1 = torch.nn.Linear(self.input_neurons,self.hidden_neurons_1)
self.hidden_layer_2 = torch.nn.Linear(self.hidden_neurons_1,self.hidden_neurons_2)
self.output = torch.nn.Linear(self.hidden_neurons_2, self.output_neurons)

def forward(self, input):
        print(input.isnan().any()) # False
        H1 = self.hidden_layer_1(input)
        H1 = self.ReLU(H1)
        H2 = self.hidden_layer_2(H1)
        H2 = self.ReLU(H2)
        final_inputs = self.output(H2)
        # not applying activation on final_inputs because CrossEntropy does that
        return final_inputs

loss = self.CrossEntropyLoss(output, target)
I am using SGD optomizer with LR = 1e-2. I don’t understand why loss becomes nan after 4-5 iterations of the epoch. Previously, when I was using just one hidden layer the loss was always finite.
When I use sigmoid instead of relu, loss stays finite.