I just tried the nightly build and the performance is much improved - and there are no calls to _th_get_device now.
For example, before (with PyTorch 0.4.1) total results were:

CPU total: 783053.1450000006
GPU total: 875131.6999999996

and now with the nightly build:

CPU total: 411711.9779999998
GPU total: 477035.2709999999

so quite a good improvement.