The increased memory on the default device is expected due to the scattering of the larger batch from this device.
We generally recommend to use DistributedDataParallel with a single process per device as described here 2 for the best performance (and to avoid the imbalanced memory usage).