Usual loss when pretraining?
We are pretraining on our own corpus using the `pregenerate_training_data.py` and `finetune_on_pregenerated.py` scripts.

The input text to the first script follows the same format as `sample_text.txt` on the samples folder, and contains about 515000 lines of text. We run `finetune_on_pregenerated.py` for 60 epochs with the default learning rate (3e-5), 128 sequence length, 300 batch size using 8gpus. The loss got to 1.3. We used this model for a classification task and we didn't see any difference than using the original pretrained BERT. We also compared the weights of some attention layers and they are very similar.

Do you have an estimate as to what the loss should be in order to see improvements? We are also aware we should use 512 for some of the pretraining process because most of our input sequences are 512 long, but still we were expecting some kind of change.

Also, do you think it might be a problem with the small size of our corpus?

@PedroUria 

We're in a similar boat as you are. In our case,  the problem was with accuracy though.

We used `BertForSequenceClassification` on  a multi-label classification task. We've actually written a similar version of `BertForSequenceClassification` as written in `models.py` of this repository changing `CrossEntropyLoss` with `BCELogitLoss`.

Initially we faced some issues with loading pre-trained weights, but once we did sucessfully and started training ( 8 label target ), even though our loss started off very small `0.24` in first step of an epoch and went down to `0.023` we never saw accuracy on training set more than `0.71` and validation `.81` in a specific batch. In comparision we have much smaller data set than yours , actually not even close : `350 rows training` and `120 rows` evaluation. So, it's likely that the poor results we're experiencing is because of the minimal data size.

We ran for only 4 epochs though, we also used `Cyclic_LR` for scheduling with `BertAdam` optimizer with same learning rate as yours. I did create an issue looking for answers. Hope they'll respond.

<|||||>Are you finetuning on the data afterwards? If yes then I believe it is due to the very small dataset. Try to finetune on a larger dataset on a similiar task first and then finetune on your small dataset.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
