For 2. I meant leave the embedding matrix in CPU but load as many rows/embedding vetors as you can fit into GPU memory in a chunk.
I just realized, are optimizers like Adam that require storing a history of gradient updates not feasible in this case since the embedding matrix is stored in CPU? It seems like the only optimizer I could use here is SGD (without momentum)?