Just a quick comment, could it be an issue that you save inp for backward and then return the output on a different device? So, you have inp on I assume your cpu and out on cuda? And, when computing your loss, grad_output will be on the device of your loss which I assume is cuda and you mask its value by a Tensor which is on the cpu? Surely this would cause some issue?