when you call register_backward_hook it traverses the modules that the module has. When you initialize a DataParallel module, it creates a copy of each module for each GPU. I would try doing the register on the DataParallel module