That’s right. You can use a one-liner and wrap your model in nn.DataParallel or use the recommended DDP approach.
Alternatively, you could also use model sharding and split the model among all GPUs in case you are working with a huge model.