Even when I synchronize GPUs like this:
    def forward(self, input):
        if self.calculate_running:
            pctl, _ = torch.kthvalue(input.view(-1), int(input.numel() * 0.9))
            self.running = pctl
            print(' gpu {}  calculate_running: {}'.format(torch.cuda.current_device(), self.calculate_running))
            torch.cuda.synchronize()
            self.calculate_running = False
            torch.cuda.synchronize()

        return self.running

It still does not work. Strange, isn’t it?