Hi,
my testing result is
test_lr_batch  c_time 1.54s py_time 1.20s

If i change x to torch.randn(4, 4800, 160), the result is
test_lr_batch  c_time 0.57s py_time 0.24s

and it seems the difference is (somehow) significant.
(by the way, i am using pytorch 1.4 + ( NVIDIA-SMI 430.40 Driver Version: 430.40 CUDA Version: 10.1) + gcc 7.5.0 + Ubuntu 18.04.4 LTS)
I added torch.cuda.synchronize() at different locations in the function test_lr_batch and it makes no change to the results.