prophet_zhan:

Such as I can set a higher batch_size without any other change, right?


I’m not sure what “without any other change” means, but you would either have to use a data parallel approach (which needs code changes) or a model parallel (also knows as model sharding), which also needs code changes.
But yes, you can use multiple devices in PyTorch.