672 is not an issue because the total number of inputs is not multiple of 1024 if remove 672 items it will run that I feel,
you also got a representation matrix and similarity matrix.
the issue I feel is in
sim_ij and sim_ji the torch.diag, because 672 is less than 1024.
are you getting the output of sim_ij and sim_ji in the last step of epoch?