Solved by kevs in post #3 
              
              
                Thank you for the extensive answer. My problem was the missing bias unit. Rewriting the forward pass as the following made the model learn non-linear patterns: 
...
b0 = torch.tensor(np.random.randn(64))
b1 = torch.tensor(np.random.randn(3))
...
out = torch.tensor(X, requires_grad=True).matmul(w0).a…