GPT sentence log loss: average or summed loss?
>>> config = GPT2Config.from_pretrained('gpt2')
>>> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
 >>> model = GPT2LMHeadModel(config)
 >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
 >>> outputs = model(input_ids, labels=input_ids)
 >>> loss, logits = outputs[:2]

For the loss value computed for the sentence, is it an average log loss or summed log loss? I had a look at CrossEntropyLoss in torch.nn and it seems to be an average loss, but thought I'd double check.

If there are multiple sentences in the input instead (so batch size > 1), what does it return? The average logloss over all tokens in the two sentences?
Yes, it's the average<|||||>Thanks for the prompt reply. Much appreciated.