I haven’t got a response for SpanBERT yet.
However, based on this comment:



 arinaruck:

The amp does not show nans with the base model, only with the large one.


Are you seeing NaN outputs in the forward pass only (without training) after loading the pretrained weights for the “large” model?
If so, are you seeing the same invalid outputs, if you create a new, randomly initialized, model?