Hello Swapnil!



 swap:


In both cases, training loss differs by several folds as compare to validation loss. What am I doing wrong?

loss_function = nn.NLLLoss(weight=normedWeights)
...
            loss = loss_function(output.double(), y)
            train_batch_loss += F.nll_loss(output.double(), y, reduction='sum')
...
        running_loss = train_batch_loss/len(train_loader)
...
                val_loss1 = loss_function(val_outputs.double(), y_val)
                val_loss += val_loss1.item()
...
        val_total += len(y_val)
        print(f"Epoch {epoch}. LOSS:: Train_loss {loss:.4f}  Train_running_loss {running_loss:.4f}   Val_loss {val_loss:.4f}")
        print(f"Accuracy:: Train {correct/total:.4f}  Validation {val_correct/val_total:.4f}")



I don’t really understand what you are trying to do nor what your issue is.
But I have a couple of comments:
When you calculate loss and val_loss you use normedWeights in
loss_function.  But when you calculate running_loss you do not
use normedWeights.  Also, running_loss is an average – you
normalize it by dividing by len(train_loader), but val_loss is just
a sum over losses, and you don’t average it over the length of your
enumerate(val_loader) loop.
So your three loss values are normalized and/or class-weighted
differently, and are not directly comparable.


        self.fc5 = nn.Linear(16, 2)
...
        x = self.fc5(x)
        return F.log_softmax(x, dim=1)

…
nSamples = [157544, 5538] #composition of classes in my dataset for class 0 and 1, resp


As an aside, you are performing a binary (two-class) classification.
You can treat this as a multi-class classification (as you are doing), but
you will be a little bit better off if you treat it as a binary classification.
So I would recommend that you have a single output from your final
layer, get rid of the log_softmax(), and use BCEWithLogitsLoss as
your loss function.
Something like this:
       self.fc5 = nn.Linear(16, 1)
...
       x = self.fc5(x)
       return x
...
posWeight = torch.FloatTensor ([nSamples[1] / (nSamples[0] + nSamples[1])]).to(device, dtype=torch.float64)
loss_function = nn.BCEWithLogitsLoss (pos_weight = posWeight)

Good luck.
K. Frank