Cos maybe you are making the net small enough to fit in the residual space of the in-use gpu. That’s why I tell you to check if cuda_visible_devices works.
If it works you just wrongly assigned the devices