Alternatively to the context manager you could also create a device variable as explained in the device-agnostic code part 10:
args = parser.parse_args()
args.device = None
if not args.disable_cuda and torch.cuda.is_available():
    args.device = torch.device(f"cuda:{ARGS.gpu}")
else:
    args.device = torch.device('cpu')

Afterwards just pass the args.device to each to() operation.