Sure, I think everything is in this commit in data_parallel branch of my fork of original repository:

  
      github.com/rafaljanwojcik/sentence-transformers
  
  
    
  
    
  

  
    
      Merge branch 'data_parallel' of https://github.com/rafaljanwojcik/sentence-transformers into data_parallel 17
    

    
      
        committed 
        
          
        
        Jan 26, 2020
      
      

      
        
          
          rafaljanwojcik
        
        
      

      
        
          +17
          -43
        
      
    

  





  
  
    
    
  
  


Essentially, I changed these lines of code:
92        self = nn.DataParallel(self)
93        self.to(device)
...
294      device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
295      for loss_model in loss_models:
296      loss_model.to(device)

to:
300  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
301  loss_models = list(map(lambda x: nn.DataParallel(x), loss_models))
302  for loss_model in loss_models:
303      loss_model.to(device)

I also changed the CosineSimilarityLoss, but it was mainly because of different training objective I wanted to use, so it’s not nessecary for multiple GPUs. Let me know if you need help, it’s a bit messy but it works