But when training these models the bottleneck is data. The model needs to perform an action to get next values and then update. That is small amount of computation so the added speed of doing on gpu from that is lossed from slower sharing updates on gpu compared to CPU. Only by collecting a bunch of these updates and updating all at once and losing all the slow individual shared updates will it be beneficial. So yes you can do that but why if it will be no faster is what I’m trying to express.