So device is a contextmanager class so best practice is to use with “with” statement. So for example your line here:
torch.cuda.device(gpu_to_use)

You are just entering and exiting so effectively performs nothing for you.
You want to use as so:
with torch.cuda.device(gpu_to_use):
    net = model.cuda()

That way any calls to cuda will go to desired gpu as long as in with-block.
Could you possibly provide a link to full script cause can’t see whats exactly going wrong though with what you provided.