I think a long runtime is expected, since InverseMelScale uses SGD to solve the mapping.
From the docs:

Solve for a normal STFT from a mel frequency STFT, using a conversion matrix. This uses triangular filter banks.
It minimizes the euclidian norm between the input mel-spectrogram and the product between the estimated spectrogram and the filter banks using SGD.

Corresponding lines of code 2.
You could change the tolerance to stop the optimization earlier, if needed.