Multilabel classification and diverging loss
Hi,

I'm not sure I'm posting this at the right spot but I am trying to use your excellent implementation to do some multi label classification on some text. I basically adapted the run_classifier.py code to a Jupyter Notebook and change a little bit the BERT Sequence Classifier model so it can handle multilabel classification. However, my loss tends to diverge and my outputs are either all ones or all zeros.

The labels distribution in my train dataset is :

`array([ 65, 564, 108,  17,  40,  26, 306, 195,  25, 345,  54,  80, 214]) ` 

i.e. the label 1 is used 65 times, the label 2 is used 564 times etc... Each sample has between 1 and 4 labels.

I am using the Adam Optimizer on the BCEWithLogitsLoss and I am unable to figure out where the problem comes from? Should I add some weights in my loss function? Do I use it in a right way? Is my model wrong somewhere? I attach to this post a Notebook of my test. Maybe someone encountered the same problem before and could help me?

[NOTEBOOK](https://nbviewer.jupyter.org/github/nicolas-mingione/nlp/blob/master/test_bert.ipynb)

Thanks !
Hey, I am working on something similar. I feel like the original code might be incorrect. They seem to directly take the output of the model as 'loss' without applying any criteria. But I might be totally wrong. <|||||>Hey! :)

Really? What do you mean by criteria?

I tried to artificially change my dataset so that the target outputs are
[1, 0, 0,..., 0] for every sample. I wanted to see whether the model was
able to learn this dummy case. However, it fails and it predicts the exact
opposite, i.e. [0, 1, 1,.... 1]. That's why I think I must be doing
something wrong somewhere.
<|||||>Hi @nicolas-mng @zhipeng-fan Did you guys manage to get a multilabel problem to work ? Could you please share a gist ?<|||||>Hi, you should have a look here: https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d ;) 