Could we explain the disadvantage of imbalanced data mathematically? Simple setup:

*

*observed response is binary (yes/no, 0/1, positive/negative).

*use logistic regression to model the probability of the response being, say, 1: $P(Y=1|X)$.

*the MLE of the model coefficients enable us to predict the probability by $\frac{1}{1+exp(-X\hat{\beta})}$.

*By checking ROC and AUC, we find the most appropriate threshold of $p_{0}$ such that $\hat{y}=1$ if $\hat{p}>p_{0}$, otherwise $\hat{y}=0$.

In this process, could we mathematically show that $P(Y=1|X)$ is underestimated if we have 10000 0's and 100 1's? How is this bias quantified in formula?

Not an explanation but an example illustrating that strong imbalance forces dismissal of the underrepresented class.
Let $y = y_1, ..., y_N$ be a sample where $y_k = 0$ except for $y_1 = 1$. Also, assume that you have observations $x_k$ such that the true probability $p(y_k = 1 \mid x_k) = 0.5$ for all $k$.
If $m(x)$ is your model output, the MSE is given by
$$
\frac{1}{N}\sum_{k=1}^N (y_k - m(x_k))^2 = \frac{1}{N}(1 - m(x_1))^2 + \frac{1}{N}\sum_{k=2}^N m(x_k)^2
$$
As $N$ tends to infinity, the first term disappears and the optimal model becomes indistinguishable from $m(x) = 0$. What is worse, the true model ($m(x) = 0.5$) gives a very large error.
