I’m working on a synthetic gradient approach and I have this simple model:
model = nn.Sequential(OrderedDict([('lin1', nn.Linear(2, 4)),
                                   ('relu1', nn.ReLU(inplace=True)),
                                   ('ef1', EF.Identity()),
                                   ('lin2', nn.Linear(4, 3)),
                                   ('relu2', nn.ReLU(inplace=True)),
                                   ('ef2', EF.Identity()),
                                   ('lin3', nn.Linear(3, 2))
	                           ]))

EF.Identity is a custom layer that simply copies the input to the output in the forward pass. I use it to register a hook in the backward pass where I overwrite gradInput. I do this to check (by print) that gradInput of ‘ef1’ is equal to gradOutput of ‘relu1’.
model.relu1.register_backward_hook(EF.print_grad)
model.ef1.register_backward_hook(EF.hack_grad)

Then I create dummy input and target variables just to test:
y_hat = model(dummy)
loss_fn = nn.CrossEntropyLoss()
loss = loss_fn(y_hat, target)
loss.backward()

But I notice that in the backward pass, the hook on ‘relu1’ is run before the hook on ‘ef1’. Is this because of some optimization going on that is changing the op graph? If so, is there a way to deactivate this behavior? It’s really important that the backward operations are run following the order for my method.