@jmaronas it does help, but again my question is of more conceptual  point of view.
if I were to compare it to keras (or tensorflow even), all you need to do in order to work with a GPU is install the proper GPU version of tensorflow (as a backend) and it will pickup all the available cuda devices automatically, whereas in pytorch you need to shift those objects each time manually. maybe it is because of the dynamic nature of pytorch.