I wrote following testing script:
import torch
import torch.nn as nn
import time

class My_ReLU(nn.Module):

    def __init__(self):
        super(My_ReLU, self).__init__()

    def forward(self, x):

        x[x < 0] *= 0

        return x


official_relu = nn.ReLU().cuda()
my_relu = My_ReLU().cuda()

data = torch.rand(1000, 1000).cuda()

start = time.time()
for _ in range(100):
    official_relu(data)
print(time.time() - start)


start = time.time()
for _ in range(100):
    my_relu(data)
print(time.time() - start)

and the output is

0.0022401809692382812
0.04940199851989746

I don’t think this huge gap is caused by just 3 kernels vs 1 kernel.