I could recommend to update to 1.1.0 and use the .to(device, non_blocking=True) method.
Have a look at the release notes for 0.4 3 to see some necessary changes. Also the most recent release notes will be useful.