Hi, I have a machine with 4 GPUs, my program went smoothly, I did as always:
    model = model.cuda()
    model = nn.DataParallel(model) 

following that I did:
        images = images.cuda()
        target = target.cuda()
        output = model(images)

My input batch is 64, 16 on each GPU, everything as planned.
Things are getting wrong when I’m splitting my model to have 2 forwards:
        midoutput = model.module.forward1(images)
        output = model.module.forward2(midoutput)

Then although I did the exect same procedure as mentioned above, inside forward1 I’m getting a batch of 64 and not 16 as expected.
when I’m using GPUtil I’m getting:

So, it seems that although I used the same procedure, and I have 4 GPU,  only 1 GPU is working now.
Any Idea?
Thanks a lot!