I don’t know how grad and grad1 are calculated, but PyTorch will stick the the specified devices in your script and won’t change the device behind your back.
Of course this doesn’t apply to any 3rd party utilities, which explicitly could push data to the CPU to save GPU memory, but I also don’t know if you are using any of these.
Could you describe your use case a bit and post a minimal code snippet, which would reproduce this issue?