I am using a pretrained SpanBERT 3 and loading it with transformers:
from transformers import BertConfig, BertModel


config = BertConfig.from_pretrained(args.model_path)
self.bert_qa_encoder = BertModel.from_pretrained(args.model_path, config=config)

The amp does not show nans with the base model, only with the large one.
Is it possible that the pretrained weights are already overflowing the fp16? Then is weight clipping before training the solution?