My code is structured like this:
for epoch in range(args.max_epochs):
    model.train(True)
    ...
    for batch_idx, input in enumerate(train_loader):
        ...
    model.eval()
    testframe = next(iter(test_loader))
    sample_frame(model, testframe)

Somehow when generating samples, the batch the generated frames are conditioned on, is the same everytime.