I think you need to port https://github.com/torch/nn/blob/master/MultiLabelSoftMarginCriterion.lua 143 to pytorch. You just need to write the forward pass in a function, and the rest should be handled by autograd