I got down to this small code snippet:
for i in range(100000):
    print(i)
    a = torch.empty(2000, dtype=torch.uint8, device='cuda').random_(2)
    a[0] = 1
    b = a.clone()
    b[:-1] = b[1:]
    b[-1] = 1

    if a.sum().cpu().item() != b.sum().cpu().item():
        break

The error occurs often at index 1023.
I’m no CUDA expert, but maybe it could be related to some kernel size etc.
@richard, @SimonW, @colesbury  Do you guys have any idea, where I could start debugging / looking into the code?