Okay, this took a while to figure out (although not completely). The problem being: the paper says one thing but the code does something else . Take a look at the other paper by the same authors (at the very bottom of the README.md in the link you posted) and could probably explain why they coded the loss function slightly different from the paper (in fact they call it the rank loss function). I hope this helps.