Oh I solved this problem with model sharding!
I splitted my embedding layer to 2 pieces with differnet size
and allocated them to each gpus
and ran forward operation on gpu who has smaller piece of embedding layer
and it works!