I think this PR 51 and this issue 39 are related to my question:

ATen currently does not have any concept of a “device” beyond CPU vs. CUDA (i.e. the “ordinal” part is missing from ATen’s model). I’m working on changing this right now, so you can expect this to be fixed soon