Have you checked that the gradient is correct for the non-combined ESM transformer on its own using your finite differencing method? Its possible that for large models the input is unstable at the point you sampled.