(PyTorch 0.4.0) Yes, when I apply your benchmarking for forward pass (as in your code) gives similar time for relu and custom rule (0.014 vs 0.014).
However, the time is slower for custom relu in both forward and backward. (0.051 vs 0.218). It seems like the backward pass takes very long. For (PyTorch 0.3.1.post2), the time for relu and custom relu in both forward and backward is (0.058 vs 0.056) with the below benchmarking…Can you please help me on this? Thank you.
    torch.cuda.synchronize()
    tm = time.time()
    output = model(input_var)

         
    loss = criterion(output, target_var)
    loss.backward()
    
    
    torch.cuda.synchronize()
    print(time.time() -tm)