I did some digging into the codebase I’m using.
The only explicit backward call is here: https://github.com/pytorch/fairseq/blob/master/fairseq/trainer.py#L425 5
where it’s actually just a wrapper that eventually calls https://github.com/pytorch/fairseq/blob/8aa06aa03b596de58d106d3f55ff43e2b9aa0b80/fairseq/optim/fairseq_optimizer.py#L85 2.
I know that besides calling backward twice, accessing certain nodes in the graphs can cause this problem but I don’t see anything out of the ordinary.
I’ve done some commenting out and noticed that when this specific code segment was executed, the error appeared. (When this block was commented out the error did not appear)
This code segment is executed right before a single training step (https://github.com/ypsoh/fairseq/blob/add_pruning_routine/fairseq/trainer.py#L498 2)
                prune.global_unstructured(self.modules_to_prune,
                        pruning_method=prune.L1Unstructured if self.prune_type == 'magnitude' \
                                else prune.RandomUnstructured,
                        amount=pruning_amount)

So what I did was I removed all the retain_graph=True calls and wrapped this with a with torch.no_grad(). and the error seemed to go away. The models seem to be pruning as expected also.
Do you think this is a viable solution? Or does this hack give you some insight on what might have been causing this issue?