What I was trying to say was that this looks like expected behavior to me. The first GPU should require and utilize more memory because that’s where the gathering of the outputs and computation of the loss happens. Like it’s outlined in the graphic I shared from the other discussion.
The results the OP showed, showed the 2nd and 3rd GPU have an evenly distributed memory usage, which is exactly what I would expect.