If I’m thinking correctly… wouldn’t detach() do a good for you? (e.g.) embedding.weight.detach().
But I don’t know. Your situation might need some unusual or non-trivial solution that is more than I can suggest =\