Lowering the loss or momentum didn’t help.
I’m going to continue training this with a single gpu for now. It should take about 5 days and then I can try some more things and pick this conversation back up.