I meet a similar problem. It seems that data parallel with forward hook cannot guarantee the activation to be on the same device as the model weights. I am not sure why, but perhaps because of using dictionary to store the activation, which is not ordered. Is there any suggestion on how to solve this? Thanks.