Looking at this first line from the output of profile:
Name                                    CPU time    CUDA time      Calls        CPU total       CUDA total
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
_th_get_device                     79.467us     77.472us             1         79.467us         77.472us

I’m seeing lots of calls to this _th_get_device function. In fact, when I post-processed the output with a python script I found that of the top 5 items on GPU time, it was the 2nd highest right after mm (which I assume is matrix multiplication):

MmBackward: GPU time: 135000.3349999999
add: GPU time: 143903.06000000058
ThAddmmBackward: GPU time: 157247.46000000002
_th_get_device: GPU time: 220894.87300000023
mm: GPU time: 260056.3770000001

Is there any way of reducing this, or is this just the nature of the beast? I don’t find a lot of information about what _th_get_device is actually doing, but assume from the name that it’s probably something to do with the CPU making a request to the GPU, in which case, it’s probably not something that can be optimized much, but is there some way to reduce the number of these calls?
Also, is there somewhere to lookup what the various function calls are doing?