Hmmm, that definitely seems like it could be the problem!  Right now I am just calling DataParallel on the whole network, while I need to do the backward hook on individual layers.  I don’t seem to be able to do net.module.layerID.register_backward_hook.  I imagine the net.module part of that is the problem.  Do I need to call DataParallel on each layer individually or is there a better way to tell a dataparallel I just want to call the backward hook on one of its submodules?
90% sure the above solved the problem calling dataparallel on each layer individually