I have a workstation with 3 gpus. Whenever I run a model on the device other than gpu:0 (say gpu:1), the model still allocates some additional memories on gpu:0 (from my observation, it varies between 600M to 900M, which seems depend on the model I am training). I call it additional memories because when I run the same model on the gpu:0, this size of the memories won’t be allocated.
The model can run, but the problem is a little annoying. Does anyone knows what’s going on here.