Thanks for the reply. I am double-checking the numerical issue. Currently, it seems like tensor broadcasting introduces a non-negligible error, an entry is expected to be zero, but we got something like 7E-7.
The loss should be greater than or equal to 0, but sometimes got a very small negative value. I haven’t figured out the reason.