Hi,
I want to train a language model with various batch sizes, esp. batch size = 1. As expected, it takes much longer time to train one epoch but at the same time the gpu utility is very low. Can I speed up  this process by data parallelism using more workers (?) on the same gpu card ? (Maybe something similar to GPU version of “Hogwild” with data parallelism ?)
Thanks.