Yup, 3 GPUs which are recognized by nvidia-smi and pytorch.
When I add:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() > 1:
    print("data parallel")
    model = torch.nn.DataParallel(model,  device_ids=[0, 1, 2])
    model.cuda()
model.to(device)

to my code but not the part in my model declaration (self.hidden_1 = nn.DataParallel(self.hidden_1)) my code runs fine but does not use all 3 GPUs, just one.
The sizes on those lines are:
torch.Size([65])
torch.Size([128])