Hi, i just add a pass in TorchScript IR to convert BertLayer to fastertransformer Encoder, however i find model is slow after convert to TorchScript. I get Nvprof result and find a time consuming activity:
Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:   57.50%  1.49484s     25200  59.319us  3.2000us  151.55us  _ZN2at6native27unrolled_elementwise_kernelIZZZNS0_21copy_device_to_deviceERNS_14TensorIteratorEbENKUlvE0_clEvENKUlvE2_clEvEUlfE_NS_6detail5ArrayIPcLi2EEE16OffsetCalculatorILi1EjESC_NS0_6memory15LoadWithoutCastENSD_16StoreWithoutCastEEEviT_T0_T1_T2_T3_T4_

I watched my final TorchScript IR, and i guess it’s reason is each time it runs it will do aten::contiguous several times, like:
%1752 : Float(*, *, requires_grad=1, device=cuda:0) = aten::contiguous(%1153, %21)

aten::contiguous is needed for Tensors which will be send to custom op because they will be convert by .transpose(-1, -2) first,  but aten::contiguous seems time consuming. So is there any way that i can convert model weights to constant in TorchScript IR so that aten::contiguous(weights) will be convert to Constant Tensor, or if i can do something to avoid aten::contiguous? Thankyou very much!