Hello all,
I have a server with 8 x GeForce GTX 1080 Ti and I am trying to implement distributed training based on this github collab https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py 1
This was meant for imagenet, but I made a few modifications for cifar10:

I had to eliminate this function, because I couldn’t make it work with cifar10, so I skipped it

def fast_collate(batch, memory_format):
imgs = [img[0] for img in batch]
targets = torch.tensor([target[1] for target in batch], dtype=torch.int64)
w = imgs[0].size[0]
h = imgs[0].size[1]
tensor = torch.zeros( (len(imgs), 3, h, w), dtype=torch.uint8).contiguous(memory_format=memory_format)
for i, img in enumerate(imgs):
    nump_array = np.asarray(img, dtype=np.uint8)
    if(nump_array.ndim < 3):
        nump_array = np.expand_dims(nump_array, axis=-1)
    nump_array = np.rollaxis(nump_array, 2)
    tensor[i] += torch.from_numpy(nump_array)
return tensor, targets



I changed the model to Efficient Net from https://www.kaggle.com/hmendonca/efficientnet-cifar-10-ignite 1


Batch size is 125 and Image size 224. It is not something excessive for 8 GPUs


I train and I get this error on ALL GPUs:


RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 10.92 GiB total capacity; 10.33 GiB already allocated; 43.50 MiB free; 43.60 MiB cached)
Traceback (most recent call last):
I believe batch size should be divided between the 8 GPUs and if that is right why they’re overflowing or how can I fix it?..I don’t see the advantage of using apex if I am still getting this kind of errors.
Thanks in advance