I’ve been struggling with this problem as well, and after searching around I figured the first approach of this answer might be the right way to “split to two embedding layers”.

  
      stackoverflow.com
  
  
      
    
  

  Is it possible to freeze only certain embedding weights in the embedding layer in pytorch? 119



  python, nlp, pytorch, word-embedding



  
  answered by
  
    Szymon Maszke
  
  on 09:55PM - 01 Mar 19 UTC


  
  
    
    
  
  


Hope it helps.