Hey,
I get the feeling that I might miss something about the perfomance and speed and memory issues using huggingface transformer. Since,  I like this repo and huggingface transformers very much (!) I hope I do not miss something as I almost did not use any other Bert Implementations. Because I want to use TF2 that is why I use huggingface.
Now, I would like to speed up inference and maybe decreasing memory usage.
As I am native tensorflow user, I have no experience with the pytorch models at all.

So is it possible that the pytorch models are more performant and more efficient than the tf models?
*How can I speed up inference ? For encoding 200 sentences pairs on my cpu it takes 12 seconds.
So is it more feasible to use pytorch models for making inference or even training?
Are there any memory usage differences?
*So, why is bert-as-a-service more performant and faster (as it looks like) I hope I can test this?

I ask because I stumpled over here:

  
      github.com/huggingface/transformers
  
  
    
  
	  
  

  
    
      Difference between this repo and bert-as-service 19
    

    
      
        opened 
        
          
        
        Apr 14, 2019
      
      

        
          closed 
        
          
        
        Jun 23, 2019
      
        

      
        
          
          tcqiuyu
        
      
    
  



  Hi,
I wondered if anybody knows the difference between the BertModel of this repo and bert-as-service.

I cannot get the same result between...



    Discussion
    wontfix


  
  
    
    
  
  



  
    
    
    How to ensure fast inference on both CPU and GPU with BertForSequenceClassification? Beginners
  
  
    Hi! 
I’d like to perform fast inference using BertForSequenceClassification on both CPUs and GPUs. 
For the purpose, I thought that torch DataLoaders could be useful, and indeed on GPU they are. 
Given a set of sentences sents I encode them and employ a DataLoader as in 
encoded_data_val = tokenizer.batch_encode_plus(sents, 
                                               add_special_tokens=True, 
                                               return_attention_mask=True, 
                        …
  


Some advices for better usage (for deployment) are very appreciated.