Something I think I have to mention is that, setting model.train or model.eval won’t effect on autograd. In fact, for instance, dropout layers have to be turned off in eval mode. So, you have to use model.eval but if you do not set required_grad= off, it still affects training and computes this values.
In you case, you do not need to put model 2 in eval mode but you have to turn gradients off (if you don’t it will affect the model1’s parameters too.)
Here is the new snippet that uses model2 on model1’s input:
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer = nn.Conv2d(3, 3, 3, 1, 1)
    def forward(self, x):
        x = self.layer(x)
        return x

model2 = vgg16_bn(pretrained=True).eval()
for param in model2.parameters():
    param.requires_grad = False
model1 = Model()
model1.train()


criterion = nn.L1Loss()
optimizer = optim.SGD(model1.parameters(), lr=0.001, momentum=0.9)

running_loss = 0.0
for i in range(3):
    x = torch.randn(5, 3, 256, 256)
    # zero the parameter gradients
    optimizer.zero_grad()

    # forward
    # track history if only in train
    with torch.set_grad_enabled(True):
        outputs = model1(x)
        outputs = model2(outputs)
        loss = criterion(outputs, torch.ones(outputs.shape))

        # backward + optimize only if in training phase
        loss.backward()
        optimizer.step()

    # statistics
    running_loss += loss.item()
    print(running_loss)