Setting op = torch.zeros(size) #zero_initializations does not have anything to do with the timing. It is the network architecture where output of the model is fed back as input along with other input. For the very first time it is initialized as zeros.