Also there is a special case worth mentioning, when you use code with torch.nn.DataParallel(model) or other methods involving multiprocessing or multi-GPU, you want to do that single iteration of model.forward() before model.cuda() and related codes for distributed training. Otherwise the new initialized tensor won’t be actually assigned to the attribute you want.