The Softmax vs LogSoftmax that I am talking about thought is not in the loss function - it’s the last layer in the net, after the LSTM.

Say you have the generic setup
    def forward(self, x):
        out = self.linear_1(x)
        ...
        out = F.relu(out)
        logits = self.linear_out(out)
        probas = ACTIVATION(logits, dim=1)
        return logits, probas

and then your training:
for epoch ...:
    for minibatch ...:

        logits, probas = model(features)
        cost = COST_FN(logits, targets)
        optimizer.zero_grad()
                cost.backward()
        optimizer.step()

where ACTIVATION would be e.g., F.softmax or F.log_softmax (where F is torch.nn.functional)

I am not sure I understand your question, but it’s ok. I believe I have the answer for my original question based on the comments above. Thank you.

So, when I understand correctly, @ptrblck  the question was whether you are passing logits or probas to COST_FN. Mathematically/conceptually, it would make sense to have this be probas that is passed to COST_FN (where COST_FN is e.g., CrossEntropy/F.cross_entropy) but CrossEntropy applies log_softmax itself. So, feeding softmax or log_softmax values to CrossEntropy, although it sounds correct based on how this functions are named, would cause weird results because of applying these fnctions essentially twice then.
It may sound super trivial and is mentioned in the documentation, but I am just pointing it out because I made that mistake some time ago when I started using PyTorch and was spending quite some time with gradient checking, figuring out what was going on