Let's load our model
model = BertForSequenceClassification.from_pretrained("bert-base-uncased")
If you used to have this line in pytorch-pretrained-bert:
loss = model(input_ids, labels=labels)
Now just use this line in 🤗 Transformers to extract the loss from the output tuple:
outputs = model(input_ids, labels=labels)
loss = outputs[0]
In 🤗 Transformers you can also have access to the logits:
loss, logits = outputs[:2]
And even the attention weights if you configure the model to output them (and other outputs too, see the docstrings and documentation)
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", output_attentions=True)
outputs = model(input_ids, labels=labels)
loss, logits, attentions = outputs
```
Serialization
Breaking change in the from_pretrained()method:
Models are now set in evaluation mode by default when instantiated with the from_pretrained() method. To train them don't forget to set them back in training mode (model.train()) to activate the dropout modules.
