Hi, I am also building a BERT model with a binary classifier. So my output logits size is [nbatch, 1].
Since my sample is really imbalanced: for 1000 cases,  positive (class 1) vs. negative (class 0) portion will be [0.101521, 0.898479].
When i am searching around about applying weights for nn.BCEWithLogitsLoss. I came across your another reply:

  
    
    
    Unclear about Weighted BCE Loss 
  
  
    If you would like to use the weight as a class weight, I assume you have a binary target, i.e. it contains only zeros and ones. 
In that case I would create a weight tensor and just multiply it with your unreduces loss. Here is a small example: 
weight = torch.tensor([0.1, 0.9])
weight_ = weight[y.data.view(-1).long()].view_as(y)
criterion = nn.BCELoss(reduce=False)
loss = criterion(output, y)
loss_class_weighted = loss * weight_
loss_class_weighted = loss_class_weighted.mean()

Let me know, if …
  


Where you were using this method:
weight = torch.tensor([0.1, 0.9])
weight_ = weight[y.data.view(-1).long()].view_as(y)
criterion = nn.BCELoss(reduce=False)
loss = criterion(output, y)
loss_class_weighted = loss * weight_
loss_class_weighted = loss_class_weighted.mean()

I modified it a little bit to match mine case:
weight = torch.tensor([0.101521, 0.898479])  # hard code from entire training dataset
pos_weight = weight[labels.data.view(-1).long()].view_as(labels)
loss_fct = nn.BCEWithLogitsLoss(pos_weight=pos_weight).cuda() # have to add the .cuda() to avoid an error
loss = loss_fct(logits, labels)

Since my batch size is 16, the pos_weight here has size 16. The model is running, but it didn’t improve my recall rate for the positive cases so far.
Then I came across your this post. Where your example make weight a single input: 10.
Seems like if i only pass one value instead of nbatch for weights, the model runs too. But i am confuse which one should i follow? Should weights size be nbatch or just one number?
Also, I want to give some example results from models. If I set epochs = 2 (When epochs = 1, my recall for positive case is always 0 on validation step).
If i don’t apply weights for the loss, my first epoch:
2020-09-23 00:36:17,114 - utils - INFO - ======== Epoch 1 / 2 ========
2020-09-23 00:36:17,114 - utils - INFO - Training...
2020-09-23 00:37:13,784 - utils - INFO - | epoch   1 |   100/ 1320 batches | lr 2.886e-05 | loss 0.354 | Elapsed 0:00:56
2020-09-23 00:38:10,029 - utils - INFO - | epoch   1 |   200/ 1320 batches | lr 2.773e-05 | loss 0.320 | Elapsed 0:01:52
2020-09-23 00:39:06,261 - utils - INFO - | epoch   1 |   300/ 1320 batches | lr 2.659e-05 | loss 0.305 | Elapsed 0:02:49
2020-09-23 00:40:03,269 - utils - INFO - | epoch   1 |   400/ 1320 batches | lr 2.545e-05 | loss 0.319 | Elapsed 0:03:46
2020-09-23 00:41:00,155 - utils - INFO - | epoch   1 |   500/ 1320 batches | lr 2.432e-05 | loss 0.288 | Elapsed 0:04:43
2020-09-23 00:41:57,041 - utils - INFO - | epoch   1 |   600/ 1320 batches | lr 2.318e-05 | loss 0.290 | Elapsed 0:05:39
2020-09-23 00:42:54,411 - utils - INFO - | epoch   1 |   700/ 1320 batches | lr 2.205e-05 | loss 0.317 | Elapsed 0:06:37
2020-09-23 00:43:51,109 - utils - INFO - | epoch   1 |   800/ 1320 batches | lr 2.091e-05 | loss 0.316 | Elapsed 0:07:33
2020-09-23 00:44:47,656 - utils - INFO - | epoch   1 |   900/ 1320 batches | lr 1.977e-05 | loss 0.278 | Elapsed 0:08:30
2020-09-23 00:45:45,236 - utils - INFO - | epoch   1 |  1000/ 1320 batches | lr 1.864e-05 | loss 0.285 | Elapsed 0:09:28
2020-09-23 00:46:41,736 - utils - INFO - | epoch   1 |  1100/ 1320 batches | lr 1.750e-05 | loss 0.260 | Elapsed 0:10:24
2020-09-23 00:47:39,109 - utils - INFO - | epoch   1 |  1200/ 1320 batches | lr 1.636e-05 | loss 0.316 | Elapsed 0:11:21
2020-09-23 00:48:36,184 - utils - INFO - | epoch   1 |  1300/ 1320 batches | lr 1.523e-05 | loss 0.260 | Elapsed 0:12:19
2020-09-23 00:48:47,433 - utils - INFO - 
2020-09-23 00:48:47,434 - utils - INFO -   Training epoch took: 0:12:30
2020-09-23 00:48:47,434 - utils - INFO - 
2020-09-23 00:48:47,434 - utils - INFO - Validating...
2020-09-23 00:49:21,259 - utils - INFO - 
2020-09-23 00:49:21,259 - utils - INFO - | loss 0.019 | Elapsed 0:00:33
2020-09-23 00:49:21,267 - utils - INFO -   
              precision    recall  f1-score      support
0.0            0.893408  1.000000  0.943703  2087.000000
1.0            0.000000  0.000000  0.000000   249.000000
accuracy       0.893408  0.893408  0.893408     0.893408
macro avg      0.446704  0.500000  0.471852  2336.000000
weighted avg   0.798177  0.893408  0.843112  2336.000000

If i use the first method: I pass weights with nbatch size when calculating the loss
2020-09-23 19:15:53,890 - utils - INFO - ======== Epoch 1 / 2 ========
2020-09-23 19:15:53,890 - utils - INFO - Training...
2020-09-23 19:16:50,893 - utils - INFO - | epoch   1 |   100/ 1320 batches | lr 2.886e-05 | loss 0.330 | Elapsed 0:00:57
2020-09-23 19:17:47,841 - utils - INFO - | epoch   1 |   200/ 1320 batches | lr 2.773e-05 | loss 0.299 | Elapsed 0:01:53
2020-09-23 19:18:43,978 - utils - INFO - | epoch   1 |   300/ 1320 batches | lr 2.659e-05 | loss 0.284 | Elapsed 0:02:50
2020-09-23 19:19:41,488 - utils - INFO - | epoch   1 |   400/ 1320 batches | lr 2.545e-05 | loss 0.296 | Elapsed 0:03:47
2020-09-23 19:20:38,118 - utils - INFO - | epoch   1 |   500/ 1320 batches | lr 2.432e-05 | loss 0.272 | Elapsed 0:04:44
2020-09-23 19:21:35,336 - utils - INFO - | epoch   1 |   600/ 1320 batches | lr 2.318e-05 | loss 0.271 | Elapsed 0:05:41
2020-09-23 19:22:32,712 - utils - INFO - | epoch   1 |   700/ 1320 batches | lr 2.205e-05 | loss 0.290 | Elapsed 0:06:38
2020-09-23 19:23:29,436 - utils - INFO - | epoch   1 |   800/ 1320 batches | lr 2.091e-05 | loss 0.291 | Elapsed 0:07:35
2020-09-23 19:24:26,124 - utils - INFO - | epoch   1 |   900/ 1320 batches | lr 1.977e-05 | loss 0.258 | Elapsed 0:08:32
2020-09-23 19:25:23,942 - utils - INFO - | epoch   1 |  1000/ 1320 batches | lr 1.864e-05 | loss 0.265 | Elapsed 0:09:30
2020-09-23 19:26:20,755 - utils - INFO - | epoch   1 |  1100/ 1320 batches | lr 1.750e-05 | loss 0.253 | Elapsed 0:10:26
2020-09-23 19:27:18,639 - utils - INFO - | epoch   1 |  1200/ 1320 batches | lr 1.636e-05 | loss 0.296 | Elapsed 0:11:24
2020-09-23 19:28:15,900 - utils - INFO - | epoch   1 |  1300/ 1320 batches | lr 1.523e-05 | loss 0.243 | Elapsed 0:12:22
2020-09-23 19:28:27,048 - utils - INFO - 
2020-09-23 19:28:27,048 - utils - INFO -   Training epoch took: 0:12:33
2020-09-23 19:28:27,048 - utils - INFO - 
2020-09-23 19:28:27,048 - utils - INFO - Validating...
2020-09-23 19:29:03,972 - utils - INFO - 
2020-09-23 19:29:03,972 - utils - INFO - | loss 0.017 | Elapsed 0:00:36
2020-09-23 19:29:03,980 - utils - INFO -   
              precision    recall  f1-score      support
0.0            0.893408  1.000000  0.943703  2087.000000
1.0            0.000000  0.000000  0.000000   249.000000
accuracy       0.893408  0.893408  0.893408     0.893408
macro avg      0.446704  0.500000  0.471852  2336.000000
weighted avg   0.798177  0.893408  0.843112  2336.000000

If I pass the weights with the second method: only one value:
2020-09-23 19:51:56,192 - utils - INFO - ======== Epoch 1 / 2 ========
2020-09-23 19:51:56,192 - utils - INFO - Training...
2020-09-23 19:52:53,044 - utils - INFO - | epoch   1 |   100/ 1320 batches | lr 2.886e-05 | loss 1.290 | Elapsed 0:00:56
2020-09-23 19:53:49,731 - utils - INFO - | epoch   1 |   200/ 1320 batches | lr 2.773e-05 | loss 1.212 | Elapsed 0:01:53
2020-09-23 19:54:45,798 - utils - INFO - | epoch   1 |   300/ 1320 batches | lr 2.659e-05 | loss 1.153 | Elapsed 0:02:49
2020-09-23 19:55:43,080 - utils - INFO - | epoch   1 |   400/ 1320 batches | lr 2.545e-05 | loss 1.128 | Elapsed 0:03:46
2020-09-23 19:56:39,890 - utils - INFO - | epoch   1 |   500/ 1320 batches | lr 2.432e-05 | loss 1.150 | Elapsed 0:04:43
2020-09-23 19:57:36,857 - utils - INFO - | epoch   1 |   600/ 1320 batches | lr 2.318e-05 | loss 1.102 | Elapsed 0:05:40
2020-09-23 19:58:33,921 - utils - INFO - | epoch   1 |   700/ 1320 batches | lr 2.205e-05 | loss 1.146 | Elapsed 0:06:37
2020-09-23 19:59:30,677 - utils - INFO - | epoch   1 |   800/ 1320 batches | lr 2.091e-05 | loss 1.154 | Elapsed 0:07:34
2020-09-23 20:00:27,239 - utils - INFO - | epoch   1 |   900/ 1320 batches | lr 1.977e-05 | loss 1.062 | Elapsed 0:08:31
2020-09-23 20:01:24,611 - utils - INFO - | epoch   1 |  1000/ 1320 batches | lr 1.864e-05 | loss 1.081 | Elapsed 0:09:28
2020-09-23 20:02:21,271 - utils - INFO - | epoch   1 |  1100/ 1320 batches | lr 1.750e-05 | loss 1.075 | Elapsed 0:10:25
2020-09-23 20:03:18,690 - utils - INFO - | epoch   1 |  1200/ 1320 batches | lr 1.636e-05 | loss 1.198 | Elapsed 0:11:22
2020-09-23 20:04:15,625 - utils - INFO - | epoch   1 |  1300/ 1320 batches | lr 1.523e-05 | loss 1.015 | Elapsed 0:12:19
2020-09-23 20:04:26,809 - utils - INFO - 
2020-09-23 20:04:26,809 - utils - INFO -   Training epoch took: 0:12:30
2020-09-23 20:04:26,809 - utils - INFO - 
2020-09-23 20:04:26,809 - utils - INFO - Validating...
2020-09-23 20:05:00,461 - utils - INFO - 
2020-09-23 20:05:00,461 - utils - INFO - | loss 0.065 | Elapsed 0:00:33
2020-09-23 20:05:00,468 - utils - INFO -   
              precision    recall  f1-score      support
0.0            0.943584  0.817441  0.875995  2087.000000
1.0            0.278409  0.590361  0.378378   249.000000
accuracy       0.793236  0.793236  0.793236     0.793236
macro avg      0.610997  0.703901  0.627187  2336.000000
weighted avg   0.872681  0.793236  0.822953  2336.000000

As you can see, with the second weights method, the loss per 100 batches goes above 1. However, even with first epoch, my recall rate is much higher.
After finishing 2 epochs, weights version 1 has a slightly higher increase on precision (5%), while recall stay same ~10%, comparing to unweighted one.
With the second weights version, my recall has been improved to ~40% while the precision also decreased from 90% to 33%.
Summary of my questions:

Can you explain what’s the difference between the first method and the second weighting method?
The goal of my model is to have a high precision for the positive case, however, i also want to make sure the recall rate is not too low. If both weighting methods work, what direction should I go?
Do you have an idea of why using unweighted loss or weight version 1, my first epoch always have recall rate of 0?

Thank you so much in advance!