Thanks @ptrblck
I did collect the log for the network so as to see what activation function Pytorch is using internally, but it doesn’t provide any information about the activation layer. However, I also verified that the other network which uses only RELU activation, its activation details were available. So, not sure why it hided leaky RELU’s activation detail. Also, I am using Google Colab, so not sure if that’s causing any bottleneck.
Regarding creating own kernel to implement leaky RELU, does it have any performance impact? Does CuDNN creates any internal pipelines or buffers to speed up the process and introducing own kernel will impact it?