I’m not sure how W and the matrix norm are defined in the paper.
If they are using the Frobenius norm, you cannot use nn.MSELoss:
w = torch.matmul(model.fc3.weight.t(),model.fc3.weight)
i = torch.eye(w.size(0))

# Frobenius norm
0.5 * (w - i).norm(p=2)
((w - i)**2).sum()**0.5 * 0.5

# MSE
criterion = nn.MSELoss()
criterion(w, i)
((w - i)**2).mean()

But as I said, I’m not familiar with the paper so I might misunderstand the notation.
The first loss looks alright.