Solved by priceluke in post #9 
              
              
                Found the issue… 
layer.weight = torch.nn.Parameter(torch.from_numpy(temp))

should have been 
layer.weight.data = torch.nn.Parameter(torch.from_numpy(temp))

Hope this helps someone in the future