That’s a good point. Currently I am struggling to test it though cause I can’t find a reference to the output of the embedding layer. I am using huggingface’s BERT model, and the call is
embedding_layer = self._model.bert_model.embeddings.word_embeddings.weight
embedding_gradients_auto = torch.autograd.grad(loss, embedding_layer,create_graph=True)

self._model.bert_model.embeddings.word_embeddings above is just an nn.Embedding module as shown here: link to huggingface’s code 4. Currently it is taking the gradient w.r.t. the weight of the nn.Embedding. How exactly do I reference the output? Right now I am gonna try to use forward hook.