I updated the code so I create x, y the way you recommended:
https://colab.research.google.com/drive/1iGZJrNGu507V2QmpSEO6tQpCK61sC8jD 3
Now, there are two implementations of my custom layer’s backward:
A)
x, weight, bias = ctx.saved_tensors
return grad_out @ weight, grad_out.t() @ x, grad_out.sum(dim=0)

B)
return None, None, None

On my machine:
A
vanilla forward: 0.0000576, vanilla backward: 0.0001834
custom  forward: 0.0000698, custom  backward: 0.0002252
B
vanilla forward: 0.0000573, vanilla backward: 0.0001784
custom  forward: 0.0000685, custom  backward: 0.0001551
So, on my machine doing nothing in backward leads to faster execution (the difference looks statistically significant; I am surprised that on your machine matrix operations perform as fast as doing nothing).
On Colab the option B is also much faster than A.