I am trying to train a Text to Image DCGAN using skipthoughts
For the pytorch skipthoughts port I am using this library 2
Currently to convert my text to embeddings I am making use of the function I wrote below
def convert_text_to_embeddings(batch_text):
  tokenized_labels, lengths = tokenizer.convert_batch_sentences_to_indices(batch_text)
  # Feeding batches of text to uniskip model
  uni_out = uniskip(tokenized_labels, lengths)
  # Feeding batches of text to biskip model
  bi_out = biskip(tokenized_labels, lengths)
  return torch.cat([uni_out, bi_out], dim=1).detach()

If you notice before I return, I am detaching it
This is because when I run the training step on the generator and discriminator, I get an unreachable error.
# Train Generator
generator.zero_grad()
input_noise = torch.randn(size=(current_batch_size, noise_size)).cuda()
fake_images = generator(input_noise, text_embeddings)
discriminator_fake_validity = discriminator(fake_images, text_embeddings)
generator_loss = adversarial_loss(discriminator_fake_validity, real_validity)

generator_loss.backward() # Error occurs here even with generator_loss.backward(retain_graph=True)
generator.optimizer.step()
generator_losses.append(generator_loss)

I was just wondering if I am making use of pretrained embeddings the right way, should I or should I not be calling detach? Also while creating the uniskip and biskip models I set them to eval mode
Are these steps right? Any help would be appreciated!
PS: If I have missed out on anything, or you would like to know more about how I implemented certain things (if my examples are insufficient or not properly explained) do let me know, I would try to be as clear as possible.