Sequence Classification One-Hot Encoded Data
## Environment info     
- `transformers` version: 3.3.1

### Who can help 
 albert, bert, GPT2, XLM: @LysandreJik 
 
## Information
I am using Bert and Roberta.

The tasks I am working on is:
Sequence Classification

## Problem
The model does not work with "One-Hot Encoded" data. The model only accepts a list of integers as labels which is feed into the MSELoss. This is undesired in a multi-label classification task with categorical data, because an order of the classes is induced.


Hi, if you want to implement a custom loss, you should not pass `labels` to the model, but instead retrieve the `hidden_states` and compute them as you would with any other PyTorch model.<|||||>Thanks for your fast answer. I found out that nn.CrossEntropyLoss expects class indices and does not take one-hot encoded tensors as target labels. So there is no difference between your implementation and using one-hot encoded labels.