Hey @Neuroinformatica,

the only thing that bothers me is that I had to comment the data_collator parameter, because it was throwing an error. However, with that commented, the training seems to be fine.

Ah that’s because for sequence classification tasks you need to either pad the examples during the tokenization step or do it on-the-fly with the Trainer. By default, the Trainer uses DataCollatorWithPadding 1 which is why commenting out data_collator works in your case, but fails when you try to use default_data_collator (which does no padding). You can find some extra info here: Using data collators for training and error analysis | Lewis Tunstall’s Blog 4

Evaluate the model to get the accuracy score: how can I do that?

You can pass the accuracy metric to the Trainer via the compute_metrics function:
from datasets import load_metric
import numpy as np

accuracy_score = load_metric('accuracy')

def compute_metrics(pred):
    predictions, labels = pred
    predictions = np.argmax(predictions, axis=1)
    return accuracy_score.compute(predictions=predictions, references=labels)

This will compute the accuracy during the evaluation step of training.

My assumption was that the 2 logits in the outputs value represent yes and no, so that “translating” them to probabilities would give me the answer to my question

Yes this is correct and your code makes sense. A slightly more elegant approach could be to wrap the preprocessing / inference in a custom Pipeline class, so then you can pass string inputs and get class probabilities in one go.

What is not clear to me is that if I try to ask a question on the same text which is the opposite (e.g. "do iran and afghanistan speak a different language" ) I would have expected the probabilities related to logits to be the opposite. What am I doing wrong?

I think your example might be a bit tricky because the answer is ambiguous based on the text. If you use negation like “do iran and afganistan not speak the same language” then you get a bigger weighting on the “no” class.

Speaking about your model, it is not clear to me how it works. I quickly read the paper you pointed me, but from that I see that the inputs of a model for this task are a passage ad a question. I don’t see them in your model deployment. Moreover, what do LABEL_0 and LABEL_1 represent?

Yes this seems to be a limitation from the Hub API which does not seem to be able to handle NLI tasks as far as I can tell. So right now, it treats (question, passage) inputs as independent examples that it assigns a yes/no label to. LABEL_0 and LABEL_1 represent default labels used the TextClassificationPipeline so I really should have specified in the model config the mapping from yes → 1 and no → 1. I’ll update the models when I get a bit of free time