Assuming that x is your input, and that your output y (and thus the resulting loss) is a function of x.W + b, you can force W to be diagonal by adding a constraint to your loss function: normal_loss + sum({norm(W_ij), i!=j}), (regularization on all elements of W except those on the diagonal).