That’s a good point, I am indeed using a multi-threaded data loader! Hm, I can’t see any child processes hanging around though.
EDIT: I just checked again, the displayed memory use of the cards 1 & 2 is indeed not just a nvidia-smi display issue, I can’t run anything on these cards until I clear out the memory manually