Hi, I played a little bit with torch.autograd.profiler
It seems that the dropout alone is really fast and according to the code (as far as I can understand it):
  
      github.com
  
  
    pytorch/pytorch/blob/master/aten/src/ATen/native/RNN.cpp#L1076-L1097

apply_layer_stack(const Layer<io_type, hidden_type, weight_type>& layer, const io_type& input,                  const std::vector<hidden_type>& hiddens, const std::vector<weight_type>& weights,                  int64_t num_layers, double dropout_p, bool train) {  TORCH_CHECK(num_layers == (int64_t)hiddens.size(), "Expected more hidden states in stacked_rnn");  TORCH_CHECK(num_layers == (int64_t)weights.size(), "Expected more weights in stacked_rnn");  auto layer_input = input;  auto hidden_it = hiddens.begin();  auto weight_it = weights.begin();  std::vector<hidden_type> final_hiddens;  for (int64_t l = 0; l < num_layers; ++l) {    auto layer_output = layer(layer_input, *(hidden_it++), *(weight_it++));    final_hiddens.push_back(layer_output.final_hidden);    layer_input = layer_output.outputs;    if (dropout_p != 0 && train && l < num_layers - 1) {      layer_input = dropout(layer_input, dropout_p);    }  }


  This file has been truncated. show original

  
  
    
    
  
  


The stacked standard module is doing the same thing as what I’m doing. So I don’t understand why it’s so slow.
Can it be an issue with at::dropout where the dispatch doesn’t work as expected ?
And I want to correct what I said before, it’s also true with unidirectional and without a PackedSequence. This strange behavior also happens with LSTM and GRU.
Here are the results (for 20 iterations)
std_fw: 0.115978s
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                         Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls  
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
             aten::_cudnn_rnn        99.44%        2.305s        99.56%        2.308s     115.382ms        2.336s        99.94%        2.336s     116.806ms           0 b           0 b       1.69 Gb      -1.11 Gb            20  
                  aten::empty         0.15%       3.556ms         0.15%       3.556ms      25.397us       0.000us         0.00%       0.000us       0.000us           0 b           0 b       2.80 Gb       2.80 Gb           140  
                 aten::select         0.09%       2.106ms         0.12%       2.780ms      69.504us     284.239us         0.01%     284.239us       7.106us           0 b           0 b           0 b           0 b            40  
    aten::_local_scalar_dense         0.07%       1.561ms         0.07%       1.561ms      39.035us     211.103us         0.01%     211.103us       5.278us           0 b           0 b           0 b           0 b            40  
                   aten::set_         0.05%       1.175ms         0.05%       1.175ms      58.767us     161.768us         0.01%     161.768us       8.088us           0 b           0 b           0 b           0 b            20  
                   aten::item         0.04%     858.459us         0.10%       2.420ms      60.496us     129.619us         0.01%     340.722us       8.518us           0 b           0 b           0 b           0 b            40  
               aten::rnn_tanh         0.04%     841.586us        99.70%        2.311s     115.536ms     173.881us         0.01%        2.336s     116.824ms           0 b           0 b       1.69 Gb           0 b            20  
             aten::as_strided         0.03%     673.823us         0.03%     673.823us      16.846us       0.000us         0.00%       0.000us       0.000us           0 b           0 b           0 b           0 b            40  
                  aten::fill_         0.02%     542.642us         0.02%     542.642us      27.132us     153.827us         0.01%     153.827us       7.691us           0 b           0 b           0 b           0 b            20  
                  aten::zeros         0.02%     454.993us         0.08%       1.829ms      91.436us     117.128us         0.01%     379.144us      18.957us           0 b           0 b       1.55 Mb           0 b            20  
                  aten::zero_         0.02%     421.823us         0.04%     964.465us      48.223us     108.189us         0.00%     262.016us      13.101us           0 b           0 b           0 b           0 b            20  
             aten::contiguous         0.02%     383.369us         0.02%     383.369us      19.168us      46.632us         0.00%      46.632us       2.332us           0 b           0 b           0 b           0 b            20  
    aten::cudnn_is_acceptable         0.02%     366.571us         0.02%     366.571us      18.329us      40.576us         0.00%      40.576us       2.029us           0 b           0 b           0 b           0 b            20  
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 2.318s
CUDA time total: 2.337s

split_fw: 0.0534257s
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                         Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg       CPU Mem  Self CPU Mem      CUDA Mem  Self CUDA Mem    # of Calls  
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
             aten::_cudnn_rnn        95.47%        1.015s        96.14%        1.022s      17.038ms        1.021s        95.98%        1.022s      17.037ms           0 b           0 b       1.65 Gb      -3.59 Gb            60  
                  aten::empty         1.01%      10.758ms         1.01%      10.758ms      21.516us       0.000us         0.00%       0.000us       0.000us           0 b           0 b       5.93 Gb       5.93 Gb           500  
                 aten::select         0.52%       5.570ms         0.71%       7.567ms      63.057us       6.854ms         0.64%       6.854ms      57.118us           0 b           0 b           0 b           0 b           120  
    aten::_local_scalar_dense         0.46%       4.872ms         0.46%       4.872ms      40.601us       4.817ms         0.45%       4.817ms      40.141us           0 b           0 b           0 b           0 b           120  
                 aten::stride         0.43%       4.522ms         0.43%       4.522ms      14.131us       0.000us         0.00%       0.000us       0.000us           0 b           0 b           0 b           0 b           320  
                   aten::set_         0.32%       3.446ms         0.32%       3.446ms      57.429us       3.411ms         0.32%       3.411ms      56.857us           0 b           0 b           0 b           0 b            60  
         aten::_fused_dropout         0.28%       2.979ms         0.91%       9.707ms     242.675us      11.934ms         1.12%      11.934ms     298.349us           0 b           0 b     703.71 Mb           0 b            40  
                   aten::item         0.24%       2.543ms         0.70%       7.415ms      61.789us       2.572ms         0.24%       7.389ms      61.575us           0 b           0 b           0 b           0 b           120  
               aten::rnn_tanh         0.22%       2.306ms        96.97%        1.031s      17.185ms       4.279ms         0.40%        1.031s      17.184ms           0 b           0 b       1.65 Gb           0 b            60  
             aten::as_strided         0.19%       1.996ms         0.19%       1.996ms      16.637us       0.000us         0.00%       0.000us       0.000us           0 b           0 b           0 b           0 b           120  
                  aten::fill_         0.14%       1.507ms         0.14%       1.507ms      25.113us       1.540ms         0.14%       1.540ms      25.665us           0 b           0 b           0 b           0 b            60  
                aten::dropout         0.14%       1.490ms         1.05%      11.197ms     279.935us       1.428ms         0.13%      13.362ms     334.038us           0 b           0 b     703.71 Mb           0 b            40  
                  aten::zeros         0.12%       1.321ms         0.49%       5.262ms      87.696us       2.482ms         0.23%       5.233ms      87.212us           0 b           0 b       1.55 Mb           0 b            60  
                  aten::zero_         0.12%       1.253ms         0.26%       2.760ms      45.998us       1.211ms         0.11%       2.751ms      45.843us           0 b           0 b           0 b           0 b            60  
    aten::cudnn_is_acceptable         0.10%       1.103ms         0.10%       1.103ms      18.375us       1.101ms         0.10%       1.101ms      18.356us           0 b           0 b           0 b           0 b            60  
             aten::contiguous         0.10%       1.091ms         0.10%       1.091ms      18.175us       1.086ms         0.10%       1.086ms      18.103us           0 b           0 b           0 b           0 b            60  
                     aten::to         0.07%     777.407us         0.07%     777.407us      19.435us      68.049us         0.01%      68.049us       1.701us           0 b           0 b           0 b           0 b            40  
             aten::empty_like         0.06%     676.402us         0.14%       1.451ms      36.284us       0.000us         0.00%       0.000us       0.000us           0 b           0 b     562.97 Mb           0 b            40  
-----------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 1.063s
CUDA time total: 1.064s