tueboesen:

that I consider a matrix with a diagonal of zero to be completely normal.


I think it is normal to have that. Just that taking the sqrt of that is not as much.
In general in ML, people try to avoid non-differentiable points as much as possible as it makes training much less stable