I’m not sure what kind of logging you are looking at and where it’s showing the activation and where it doesn’t. Could you explain it a bit more?



 avyz:

Regarding creating own kernel to implement leaky RELU, does it have any performance impact? Does CuDNN creates any internal pipelines or buffers to speed up the process and introducing own kernel will impact it?


I was thinking about reusing PyTorch’s internal kernel.
Since there is no leaky ReLU kernel in cudnn, there wouldn’t be any performance impact, as there wouldn’t be a cudnn baseline to compare against.