Rank 1 test on device cuda:1
after data=data.to(device,), before output=model(data) in test function,  batch_idx: 0 device: 1

This is weird. This means all model parameters are on cuda:1 and the input batch is also on cuda:1, but somehow one of the conv layers still throws device mismatch? I am not sure what happened here, but as the error suggests the mismatch occurs in cudnn_convolution, I would check if the input (x) of and the parameters the two conv layer (self.conv1 and self.conv2) match in the forward() function during testing.
BTW, two more comments on the script:

As you are only doing forward during testing, it is not necessary to use DDP there, as all comm in DDP occurs during backward.
I noticed you saving a DDP module and then load that DDP module and wrap it with another DDP module. Is this intentional? Shouldn’t mnist_train.py save model.module instead? (or use model.module to initialize DDP instances in testing)