put quantize_dynamic in fairseq-generate’s code and you will observe the change.
ref :

  
      github.com/pytorch/fairseq
  
  
    
  
	  
  

  
    
      Does Dynamic Quantization speed up Fairseq's Transfomer? 76
    

    
      
        opened 
        
          
        
        Mar 24, 2020
      
      


      
        
          
          gvskalyan
        
      
    
  



  What is your question?
As per this tutorial in torch, quantize_dynamic gives speed up of models (though it supports Linear and LSTM...



    question