Few things before trying gradient clipping:

What does your input data look like? Make sure it’s in the correct form, you would expect. Sometimes unnormalized input can cause huge loss values.
What optimizer and lr are you using?
Not sure if it’s a good idea to sum the loss values before loss.backward()





 Hari_Krishnan:

loss = cel.masked_select(mask.reshape(-1)).sum()