It’s an old conversation but I just had a similar question and found this in the doc notes on best practices within CUDA semantics 599, so thought of leaving it here in case it helps others:

This is the recommended practice when creating modules in which new tensors need to be created internally during the forward pass.

where ‘this’ refers to using one of the new_* methods for creating tensors that preserve the device context.