Not exactly a minimal example, but I think it is sufficient to reproduce:
I cloned https://github.com/pytorch/extension-cpp 8 (which is from the tutorial i linked in the question).
I opened check.py and changed line 86 from
    device = torch.device("cuda")

to
    device = torch.device("cuda:2")

I verified the cuda flag is not used in any other place to set the device of a tensor.
when I ran “python check.py --cuda forward” on the cluster with 4 GPUs I saw similar behaviour: python was using GPU 0 and 2, and the check did not pass the cuda test.  running the original version used only GPU 0 and passed both tests.
(on a side note, “python check.py --cuda backward” didn’t pass even without code change).