There are two different models in my program that share word embedding parameters and other parameters are independent. We want the word embedding parameters can be updated synchronously by both.
I have to use multi-GPUs for training because of the big training dataset.
I used multiprocessing and distributed method to utilize computation of GPU. Each model used two GPUs respectively.
However, I don’t know how to implement the share word embedding between both models that they distributed in different processes and different GPUs.
How to guarantee the word embedding parameters synchronous update ?