Yes, you are right. If you don’t want take this part into autograd process, you can wrap it into torch.no_grad() scope.
And we often use torch.no_grad() to avoid some error (i.e. modify the parameters of the network inplace). For whether it can have beneficial impacts on your embeddings, I think it depends on your use case.