I was using slurm on server and I found out that
gpu 0 in nvidia smi != gpu :0 in the codeI made following two changes
device = torch.device(“cuda:” + “1”)
encoder = torch.nn.DataParallel(encoder, device_ids=[1, 0])
and now the error says gpu 0 is out of memory
so 0 in code is actually 1 , and vice versa.
so, the only issue remains that why using 2 gpus with DataParallel goes out of memory while using 1 gpu with same data and batch size doesn’t
Does doing DataParallel requires extra memory to manage parallelism ?