Yes, and the operations run in parallel indeed. Both in forward and in backward. I’m not sure how is that not useful and what’s the problem. Data parallelism is a well known term used to describe weak scaling of these models, and this is exactly what happens in here. It’s very useful, because in some cases you need batch sizes so large, that they don’t fit on a single GPU. The behaviour is exactly what’s wanted in a lot of situations.
It doesn’t make it hard, I really don’t understand what’s bothering you. You can do hogwild training, even on GPUs if you wan. No one is stopping you. We even have a hogwild example in the examples repo.