Five observations:

for me the (i5-7500 CPU reporting for processors and a 1080Ti), 5000 loops on CUDA will be 12 seconds, but CPU much longer (500 loops in 23 seconds),
double is much slower on the GPU than float. This is why float is the standard  type in PyTorch. On (x86) CPUs, it probably doesn’t matter much,

loss = (y_pred - y).pow(2).sum().item() will take the result (living on GPU up until the sum()) and then transfer it to the CPU for .item(). This kind of synchronisation point makes the it slow. You can drop this line. For printing, I usually only to this every x iterations where x is adjusted to mean every 5-10 seconds,
your network is really small, so one would expect GPU to not provide as much advantages as for larger ones,
automatic backward is probably faster than the manual one in this case.

Best regards
Thomas