Pytorch is memory efficient, but it does not handle the devices for you per default. This is done on purpose and allows the user to use the same code on different devices and specifying the devices to use.
The feature, you requested is more likely implemented in a high-level wrapper (although I must admit, that I do not know whether any of the wrappers implemented this actual feature.