Solved by dashesy in post #3 
              
              
                I think this PR and this issue are related to my question: 

ATen currently does not have any concept of a “device” beyond CPU vs. CUDA (i.e. the “ordinal” part is missing from ATen’s model). I’m working on changing this right now, so you can expect this to be fixed soon