When I add clone() to these two Variables, it works fine.
        grad_latter_all = grad_output[:, c//3: c*2//3, :, :].clone()
        grad_swapped_all = grad_output[:, c*2//3:c, :, :].clone()

Thank you for you help.