Solved by Yanli_Zhao in post #2 
              
              
                tensor.register_hook(customHook) may work, you need to write customHook to modify grad of the tensor. 
but as far as I know customHook should be a function of grad only. For your case, you want to make customHook to be a function of grad and workerRank as well?