Hi!
I’m new here and i am writing custom activation function with learnable parameters:
How it looks:

image1314×919 85.9 KB

The formula:
x[ x < 0 ] = x[ x < 0 ] * alpha_low
x[ x > bias ] = x[ x > bias ] * alpha_high + bias
I tried several ways implementing it and here is the fastest result:
class CustomRelu(nn.Module):
	def __init__(self, alpha_low=0.01, alpha_high=0.01, bias=1.):
		super(CustomRelu, self).__init__()
		
		self.prelu_low = nn.PReLU(init=alpha_low)
		self.prelu_high = nn.PReLU(init=alpha_high)
		
		self.bias = bias
	
	def forward(self, x):
		x= -self.prelu_low(x) + self.bias
		x= -self.prelu_high(x) + self.bias
		return x

But the problem is, its still much slower than built in activation functions (about 5-10 times slower)
Is there a way how to optimize it?