Hey! Thank you for replying
This is the error log I see if I am not detaching (I set the model back to train mode)
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

In general I’d just love some advice on how to use pretrained embeddings.
I have also made use of hugging face transformers (bert) and I followed a similar approach wherein I set the model to eval mode (and output_hidden_states to True)
I then created a similar function (I referred to this)
def convert_text_to_embeddings(batch_text):
    encoded_input = tokenizer(batch_text, return_tensors='pt', padding=True, max_length=69).to('cuda')
    embedding = bert_model(**encoded_input)[2]
    output = torch.mean(embedding[-1], dim=1).squeeze()
    return output.detach()

My aim is to train a GAN based on text (Text to Image) so I just trying to clarify if my approach is the right one 
Thank you!