The GPU implementation works currently with a sorted array, such that the sorted argument is ignored.
If sorted is set to False, the CPU implementation uses std::unordered_set if the dim argument is skipped (which does not guarantee any order) or a sorted tensor and the “slow-pointer/fast-pointer” implementation, if the dim argument is specified.
What is your current use case? Maybe we could use the inverse indices to fix your use case.