@cyberwillis  Why are you assuming that I am using anaconda?
Are you experiencing the same problem or not? How you tried to replicate my results or at least run the code on a machine with multiple GPU? Is it well known pytorch behavoiur or it fails only in my case.
I hate myself for saying this, but I have not seen any reasons why should I try to build pytorch from source with Magma enabled.