I can’t find any obvious mistakes, but a lot of your methods are also undefined.
Just to understand the issue correctly:



 elbert:

However, using ‘with torch.no_grad()’ and ‘self.eval()’ simply can’t speed up my custom function module(11it/s).


Are you comparing the forward passes only?
If so, then note that using no_grad() will avoid storing the intermediate activations, but they would still need to be calculated, so I wouldn’t expect a huge speedup. However, since more memory would be free, you could increase the batch size and thus increase the throughput.



 elbert:

I happened to find that if I set the model to training mode(self.train()) first and perform backward once, the model evaluation speed will double(24it/s). Is there a standard setting method?


You haven’t shared your profiling code either, but note that the first iteration will be slower (regardless if in training or evaluation mode), since PyTorch needs to allocate memory, cudnn might profile different kernels (if torch.backends.cudnn.benchmark=True is used) etc.