To anyone who has the same problem:
grad_input[input < 0] = 0

is a slow operation. (Although it is used in the official tutorial)
Changing it to:
grad_input = grad_input * (input > 0).float()

works for me.