I got it!
I had to change the parameters to the save function to use the same signature as inference.
fastspeech2.save('text2mel', signatures=fastspeech2.inference)

For whatever reason the signature save was picking before was using dummy data from elsewhere in the code.