I’m not  sure how to properly ask this question or if I’m missing something. I am training a RNN conditioned with images and text.   In my training, I take the generated output from the rnn model, and then generate images from that.  I then compute the loss, but I cant run loss.backward()  and I dont know how to formulate the code to get the correct loss.  Here is what I mean:
    criterion = nn.L1Loss()
    for epoch in range(args.num_epochs):
      for i, (images, captions, lengths) in enumerate(data_loader):
        
        images = Variable(images, volatile=False)
        captions = Variable(captions,volatile=False)
        
        # Forward, Backward and Optimize
        decoder.zero_grad()
        encoder.zero_grad()
        features = resnet152_encoder(images)
        outputs = lstm_decoder(features, captions, lengths)
        output_features = outputs_to_image_tensor(outputs)
        loss = criterion(output_features, images)
        loss.backward()
        optimizer.step()

With that code I get error: “RuntimeError: there are no graph nodes that require computing gradients” I think I understand why, it is because in the line " output_features = outputs_to_image_tensor(outputs)" I have done a transformation to the outputs that loses the weights.  How can I do back propagation here?  Or how do I connect output_features to outputs. What I essentially want it to update outputs weights based off loss of the original images with the outputted images.  Does this make sense what Im saying?