oh interesting, lower down the code is different:
# torch.device object used throughout this script
device = torch.device("cuda" if use_cuda else "cpu")

I guess that solves the non data parallel part of my problem.