The data would still be split in its batch dimension and the forward activations would thus be shared between all devices, which are usually much larger than the input data.
However, @techkang is right and the scatter/gather ops from the default device create a memory imbalance which is why we recommend the usage of DDP (besides DDP also being faster).