Thanks for your response @ptrblck. I do not believe my inputs are the culprit - they are just indices that then look up the embedding within the in_embed layer. Do you mean to check this for at model init for all of in_embed.
In the meantime, I am currently running the model using the torch.autograd.detect_anomaly(True).
Thanks again