I don’t think the plot shows the plain ReLU, but a LeakyReLU using the default negative_slope value of 1e-2.
You can zoom in a bit and would see the tiny slope:

image895×204 19.1 KB

To reproduce the figure, run:
act = torch.nn.LeakyReLU()
x = torch.arange(-7, 7, 0.01)
out = act(x)

f = plt.figure()
plt.xlim([-7, 7])
plt.ylim([-7, 7])
plt.grid()
plt.plot(x, out)

which will create:

To better see the slope, change the default value:
act = torch.nn.LeakyReLU(negative_slope=0.5)
x = torch.arange(-7, 7, 0.01)
out = act(x)

f = plt.figure()
plt.xlim([-7, 7])
plt.ylim([-7, 7])
plt.grid()
plt.plot(x, out)