Solved by ptrblck in post #2 
              
              
                I don’t think the plot shows the plain ReLU, but a LeakyReLU using the default negative_slope value of 1e-2. 
You can zoom in a bit and would see the tiny slope: 
 [image] 
To reproduce the figure, run: 
act = torch.nn.LeakyReLU()
x = torch.arange(-7, 7, 0.01)
out = act(x)

f = plt.figure()
plt.xlim…