Hi there,
I have a model that trains on some machines but not others, I have tried running this model on 5 different machines,
machine 1: trained before, not anymore
machine 2: still trains
machine 3: has never trained properly
machine 4: still trains
machine 5: still trains
If I could I would just use machines 1,2,or 5, but machines 1 and 3 don’t have enough ram and machine 5 does not support cuda.
Only machine 4 is windows, all others are linux. I am running in a venv and have ensured that the code across all the machines is identical except where I need to remove multiprocessing from windows.
Edit, I have now found out that changing the device to CPU on machine 1 and 3 allows it to run properly, but I still don’t know why it wont work with cuda.