Hi, thanks for your help. Yes, there was overflow in gradients for fp16 like you mentioned. I ended up using utils.checkpoint and was able to fit a larger model. Could I also trouble you one more time?
I ran into this error while using utils.checkpoint:
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
It happened during the loss.backward() function. loss prints as tensor(0.1663, device='cuda:0)
My summarised code snippet looks like this:
outputs = checkpoint(self.t5, kwargs['input_ids'], kwargs['attention_mask'], None,\
                                 decoder_input_ids , kwargs['decoder_attention_mask'])
loss_fct = CrossEntropyLoss(ignore_index=-100)
loss = loss_fct(outputs[0].view(-1, outputs[0].size(-1)), lm_labels.view(-1))
...
loss.backward()

I tried following checkpoint tutorial 10 you mentioned in another thread but I couldn’t figure out what went wrong. sorry to bother you again!