I totally agree that if index is in the cpu, which it is by default, then using it to slice a cuda tensor might be a cause of slow down.  But this slowdown should have a consistent time, irrespective of the length of conv1d preceding it.
Also, in pytorch 1.8.1, even if the index was placed on cuda did not fix the issue.  I’m just wonder if I’m running across a weird corner case of extremely long kernels that is not encountered by the testing.  The fact that it behaves differently across versions is worrisome.