Then, if 3 gpus are used with batch size 30 then shouldnt they be atleast as fast as single gpu with batch size 30? And i guess there is possibility of out of memory as we load whole batch onto first gpu and nn.DataParallel() then divides equally to all gpus (out of memory when whole batch is loaded onto single gpu). Please correct me