Hello community,
I train on a cluster gpu-farm of Tesla V100s, the scheduling system will assign me to the next available gpu, I have no control over that.
One issue I realized recently is that by manual seeding both numpy and pytorch, I got consistent random numbers from numpy but different random numbers from pytorch across different GPUs. (correction: I just realized numpy random numbers are generated by CPUs.)
The manual seed to get matching random numbers passes on my local card and one particular V100 card that I could access. So I wonder if this behavior from pytorch is by design or a bug?
Thanks!