hollowcodes:

So if I would built a custom loss function in Pytorch it would need loops and if-else checks to collect or find all the same labels. Is that even differentiable then? Or can I just implement it in Pytorch and the dynamic computation graph will figure it out itself? Answers appreciated!


So you (likely, except for tiny datasets) cannot just backpropagate through everything because you’d exhaust memory before that. I’d probably look at adapting the self-supervised learning methods (like MoCo) - in a nutshell they

maximize batch size as far as they can,
use a longer history of data (without computing gradients for this bit) when computing the gradients for the other parts.

Best regards
Thomas