I have written this!!



from transformers import LongformerTokenizer, LongformerForMultipleChoice
import torch






tokenizer = LongformerTokenizer.from_pretrained(‘allenai/longformer-base-4096’)
model = LongformerForMultipleChoice.from_pretrained(‘allenai/longformer-base-4096’)






prompt = “In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced.”
choice0 = “It is eaten with a fork and a knife.”
choice1 = “It is eaten while held in the hand.”
labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1






encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors=‘pt’, padding=True)
outputs = model(**{k: v.unsqueeze(0) for k,v in encoding.items()}, labels=labels)  # batch size is 1






the linear classifier still needs to be trained
loss = outputs.loss
logits = outputs.logits



But there is only this!!
Some weights of the model checkpoint at allenai/longformer-base-4096 were not used when initializing LongformerForMultipleChoice: [‘lm_head.bias’, ‘lm_head.layer_norm.weight’, ‘lm_head.decoder.weight’, ‘lm_head.dense.weight’, ‘lm_head.layer_norm.bias’, ‘lm_head.dense.bias’]

This IS expected if you are initializing LongformerForMultipleChoice from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing LongformerForMultipleChoice from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of LongformerForMultipleChoice were not initialized from the model checkpoint at allenai/longformer-base-4096 and are newly initialized: [‘classifier.bias’, ‘classifier.weight’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

How I can fix it to show me an output??