I have a pretty large embedding matrix (pretrained and frozen) and I don’t want to copy it to each GPU when using DataParallel.
My ideal situation is the embedding matrix is on CPU, the embedded input is pinned, and the embedded input is sent to their respective GPUs when using DataParallel.
Is this possible? Or reasonable? I’m kind of at loss at the right way to handle this.