Hi @smth,
Thanks a lot for the pointers you provide. I also have other questions related, hope you can provide some information.
My project involve solving some straggler (slow worker) issues in distributed cluster. So, I want to “skip” backward calculation (gradient computing) for some layers in the network.
Say if I run the training code on a certain node in a cluster, and after I do the backward process at a certain layer(layer_10.backward -> layer_9.backward -> layer_8.backward), at this time I decided this node is too slow and I want to just simply skip calling backward step of remaining layers to avoid more time costs (e.g. by simply assign a certain value to gradients of those remaining layers rather than actually calculate the gradients) .
Is this possible in current pytorch API? Or I need to customize modules (say convolutional layer)?