Thank you for the quick reply. Does “tricky” in this scenario mean you can’t learn anything in the MLP? Do you have anything for me to read to possibly address this? Still huge thanks if there isn’t any, this seems like a tough question.
What I have on top of my head is I can probably train the MLP separately before actually training other parts of the network, using some carefully designed loss?