Dear albanD and KFrank,
thanks for your answers!
@KFrank: The idea to change the optimizer (in my case SGD) worked in my case.
In order to do so, I split up the matrix E into several components, one for each array in the parameters of the model. Then, I could manually backpropagate the gradients of all the arrays through the embedding. I then re-embedded this into the full parameter space.
Mathematically, that corresponds to multiplying the gradient with E*E.T, which is probably what you meant by “projection”.
In case you’re interested: I’m reproducing the results from this paper on the intrinsic dimension of objective landscapes: https://arxiv.org/abs/1804.08838 12
Best regards,
Leon