You might see some overhead for tiny workloads in PyTorch.
Changing the input_size and layer_width to 1000 gives:
pytorch answer = [1.0642374]
pytorch takes 16.20141328498721 s
my answer = [1.06423738]
my imlementation takes 150.2599041541107 s

I had to kill the run after a few minutes using 2000 for both variables, as numpy took too long.