Sorry por the self reply, but i found something. My problem is bert, after seeing one batc the global average pool start to be constant in all the batch. These are the global averages per batch:
Batch 1:
ensor([[[-0.0199]], [[-0.0202]],  [[-0.0123]], [[-0.0154]], [[-0.0150]], [[-0.0177]], [[-0.0165]], [[-0.0149]]], device='cuda:0', grad_fn=<ViewBackward>)

Batch 2:
tensor([[[-0.1097]],  [[-0.1097]], [[-0.1097]],  [[-0.1097]], [[-0.1097]],  [[-0.1097]],[[-0.1097]], [[-0.1097]]], device='cuda:0', grad_fn=<ViewBackward>)

Batch 3:
tensor([[[-0.1508]], [[-0.1508]], [[-0.1508]],[[-0.1508]], [[-0.1508]], [[-0.1508]], [[-0.1508]],  [[-0.1508]]], device='cuda:0', grad_fn=<ViewBackward>)

So, it seems bert its predicting constant on the batches, maybe it’s a problem of my implementation but i don’t know how to fix it. Any help will be super welcome. Thanks