I am running forward and backward on gpu and transfer gradients to cpu, then run optimizer on cpu and transfer back the updated weights to gpu. I see different performance on different machines and all are much lower than expected.