I can not speak much about the manual approach as I haven’t tried it, but regarding pin_memory=True I observe in practice that it slows done the training at about 2x (compared to False) – tested it in PyTorch 0.4.1 and 1.0 and on two independent machines (one with 1080Ti’s and one with Titan V’s). So, in practice, I abandoned using that. I remember there was a thread where someone mentioned similar observations.
So, it may well be that there’s a bug with pin_memory = True, esp. since you observe that the manual approach results in the expected speed up.