@soulitzer, that’s an interesting point, about the hooks.
What would you do if mod and mod2 have to be the same network with exactly the same parameters (as it is in my case) ? Could you think about easy way to handle it in this case?
As much as I can understand I need some variable to differ the point of using the hooks and not using them on the same parameters, right ?
like to support a flag of some specific layer, suppose I know that the last layer name is “last_layer”(and during the backward path I should see it only twice in total), so my hook should look like:
use_hook = 0
def backward_hook(x):
   if cur_layer_name == 'last_layer':
       use_hook+=1
    if use_hook < 2:
        return x/10
    else:
       return x

And that is possible if I can indeed to check through the backward path using a hook the current layer name (can I?) ?



 soulitzer:

I guess it would be the same, but a better way of doing that would just to be to clone the output y and then detach?


I need to think about that.
And another thing, correct me if I wrong, if you had two nn.Linear layers, you still have to do the hook only once because of the chain rule (and the multiplications of all the gradients according to it, correct ?)?
Or is should be done for all the differential layers (wouldn’t it be 10^{#layers_num} factor in that case) ?