if they are the first layer of your network, it’s nice to put the 50 million float numbers on CPU, get the embeddings per sequence and transfer these on the fly to appropriate GPUs.
Embedding operation is heavy sparse accesses, so it’s nicer to have it on the CPU.