Hi,
Thank you for your reply:
by second last layer I mean, layer before the first fully connected so it would be conv2 in BasicBlock1 of layer 4.
By the way I am using re-scaled cauchy distribution (kernel) (PDF) function 1/(1 + (x*x)), which I have implemented like this:
import torch

def pdf_cauchy_distribution(tensor):
    '''
    this fuction takes the output from neural netwrok's layer and implements a 
    kernal function which acts as an activation function.
    
    input:
    tensor: output of neural network's layer computation (w*x + b)
    
    output:
    also a tensor which after going to pdf cauchy distribution fucntion
    which is f(x) = 1/(1+x^2)
    '''
    
    return (1 / (1 + torch.mul(tensor, tensor)))

but when I use it like you described, it gives an error:
TypeError: cannot assign 'model.activation.pdf_cauchy_distribution' as child module 'relu' (torch.nn.Module or None expected)

Is there any functionality for linear activation functions such as the one I described offered by Pytorch?
I found this https://pytorch.org/docs/stable/distributions.html#cauchy, but I am not sure how does that serve my purpose?