@claudiomartella



 claudiomartella:

This assumes the model can fit inside of GPU memory. Is there a natural way in pytorch to run across multi-GPU a single model.


This inherently will involve some idling because of the sequential nature of the forward + backward. This is model parallelism (as opposed to data parallelism). For example, see here 162