Hi @valhalla,
I tried your code, with the only changes being that with the current version of transformers I am returning a dict, and, if I am not mistaking, I am using an AutoModelForCausalLM model with a standard bert-base model.
text = "Who was Jim Paterson ? Jim Paterson is a doctor".lower()
inputs  =  tokenizer([text],  return_tensors="pt").to(device)
input_ids = inputs["input_ids"]
# mask the token
input_ids[0][7] = tokenizer.mask_token_id
labels = inputs["input_ids"].clone()
labels[labels != tokenizer.mask_token_id] = -100 # only calculate loss on masked tokens
ret = model(
    input_ids=input_ids,
    labels=labels,
    attention_mask=inputs["attention_mask"],
    token_type_ids=inputs["token_type_ids"],
    return_dict=True
)

loss = ret.loss
logits = ret.logits

print(loss) # --> tensor(29.3720, device='cuda:0', grad_fn=<NllLossBackward>)

# calculate loss manually
import torch.nn.functional as F
loss2 = F.cross_entropy(logits.view(-1, tokenizer.vocab_size), labels.view(-1))

print(loss2) # --> tensor(20.6763, device='cuda:0', grad_fn=<NllLossBackward>)

As you can see, the results are different. What I’m trying to achieve is to calculate the perplexity of a sentence given a LM like BERT, and if I understood correctly I’ll move the label for each word in turn and then apply the ppl formula.
My question: shouldn’t the two losses be equal?