Solved by EJHyun in post #3 
              
              
                Oh I solved this problem with model sharding! 
I splitted my embedding layer to 2 pieces with differnet size 
and allocated them to each gpus 
and ran forward operation on gpu who has smaller piece of embedding layer 
and it works!