I am also trying to build a loss function based on distance of two unordered set for a different application. Also trying to figure out how to  best do this in pytorch. I’d be very interested to know if you found any good solutions! Thanks.