Under what conditions do different choices of link function for GLMs result in considerably different models? The statistical folklore I have heard is that the choice of link function usually does not considerably affect the resulting fit of a GLM. For example, usually probit regression and logistic regression give models that generate similar predicted probabilities. D. R. Cox made a similar comment in Analysis of Binary Data.
How would one mathematically describe this similarity and under what conditions it holds?

Both the Probit (normal distribution Link Function) and the Logit (Logistic distribution Link Function) are symmetrical and are not too different from each other. A good explanation (with History) can be found J.S. Cramer (2002), The Origins of Logistic Regression  https://www.econstor.eu/bitstream/10419/86100/1/02119.pdf
The non-symmetrical such as Gompit (Gompertz Ln(-Ln(1-p)) or it's negative) link functions differ more than the symmetrical link functions.
Here is a graphical comparison between the Normal CDF (Probit) vs the Logistic CDF scaled to standard deviation (using Sqrt(3)/π). Note they are very similar. The Logistic has heavier tails, and higher probability of being near the mean as can be seen by the gap ~ 1σ (ref excess Kurtosis).
From a mathematical standpoint they are highly correlated (in Plot below: R=0.997, R-squared=99.4%). Here is a linear plot for a sample size of 250 using median ranks to estimate (p) (~MR=(rank-0.3)/(n+0.4)). Note they primarily differ beyond +/- 2 Standard Deviations (σ). In a sample size of 250 that represents ~ 5 individual samples beyond 2σ in each tail.  I hope this helps.

