Hm, can you check if outputs and targets have the same shape? E.g.,
assert len(outputs.size()) == 1
assert targets.size() == outputs.size()

because of things like
In [2]: import torch                                                                                                                     

In [3]: a = torch.tensor([1., 2., 3.])                                                                                                   

In [4]: torch.nn.functional.mse_loss(a, a)                                                                                               
Out[4]: tensor(0.)

In [5]: torch.nn.functional.mse_loss(a, a.view(-1, 1))                                                                                   
Out[5]: tensor(1.3333)


I think you need to average the L2 component because by default mse_loss is averaging the loss. I.e,.
l2_reg = sum(l2_reg) ==> l2_reg = sum(l2_reg) / l2_reg.size(0)