If I want to use multiple gpus for a network should i specifically write my network in a way that it is designed to be train on multiple gpus or can i just add some comment to switch between one gpu and multiple gpus?
For example, lets say im doing this example (training a classifier: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html 281) and I used to train it on one gpu, can i easily do something to train it on multiple gpus?
If the answer is no, please consider this for as your possible future features.
Thanks