Hi, I’m trying to fine-tune BERT for entity recognition task.
While training, the train loss decreases steadily and evaluation scores increase as I expected, but suddenly at a certain point, the train loss becomes large and all evaluation scores become 0.
tempsnip962×834 16.1 KB
I think this might be vanishing/exploding gradient, but I don’t know how to fix this.
I added gradient clipping and learning rate scheduling which reduce the learning rate if the validation score does not decrease.
And I also initialized the weights of output layer for token-level classification into the xavier-uniform.
Here are specific details for my training.

encoder: bert-base-uncased
initialization of output layer: xavier uniform
batch size: 8
starting learning rate: 5e-5
scheduler: ReduceLROnPlateau

factor: 0.1
patience: 3 epochs
threshold: 1e-3


optimizer: AdamW
loss: CrossEntropy
gradient clipping: 1.0

Thank you.