Cool!  That worked for all of my tensors that are having their values be modified without the .to().  However, later in the same function there is one more tensor I am just saving directly with
self.current = grad_output[0].detach()

This works fine on one GPU but when running on multiple it tells me 'AttributeError: (module) object has no attribute ‘current’ '.  I imagine I could do a .copy_() or something, but I won’t actually know the shape until the network has been run on an image.  Thoughts?  I can also start a new thread if that would be appropriate.