Balmer proportionality How did Johannes Balmer arrive at
$$
\lambda \propto \frac{n^2}{n^2-4}, \quad (n=3,4,\dots),
$$
and then how did Rydberg mathematically derive
$$
\frac{1}{\lambda}=R\left(\frac{1}{n^2_1}-\frac{1}{n^2_2}\right)?
$$
I know $n$ stands for the shells but in the textbook, it doesn't define what $n$ is at first. Was this because Balmer did not know what shells were at that time?

I recommend reading Balmer's original paper "Notiz über die Spektrallinien des Wasserstoffs" (1885).
Balmer took the known wavelengths of the visible hydrogen spectrum
($H_\alpha$, $H_\beta$, $H_\gamma$, $H_\delta$) as measured by
Ångström with high precision.
He recognized they are related by certain fractions.
$$\begin{array}{c|c c c}
  & \lambda    \\ \hline
 H_\alpha  & 656.2 \text{ nm} &= 364.56 \text{ nm} \cdot \frac{9}{5}   &= 364.56 \text{ nm} \cdot \frac{3^2}{3^2-4} \\ \hline
 H_\beta   & 486.1 \text{ nm} &= 364.56 \text{ nm} \cdot \frac{4}{3}   &= 364.56 \text{ nm} \cdot \frac{4^2}{4^2-4} \\ \hline
 H_\gamma  & 434.0 \text{ nm} &= 364.56 \text{ nm} \cdot \frac{25}{21} &= 364.56 \text{ nm} \cdot \frac{5^2}{5^2-4} \\ \hline
 H_\delta  & 410.1 \text{ nm} &= 364.56 \text{ nm} \cdot \frac{9}{8}   &= 364.56 \text{ nm} \cdot \frac{6^2}{6^2-4}
\end{array}$$
This could be summarized in one formula.
$$\lambda=364.56 \text{ nm} \cdot \frac{n^2}{n^2-4} \quad\text{with }n=3,4,5,6$$
You see, there was no physics involved here, "only" guessing a formula which exactly fits the experimentally measured numbers.
Rydberg rewrote Balmer's formula using the reciprocal wavelength
because then it gets the simpler form of a difference between two terms.
$$\frac{1}{\lambda}=\frac{1}{91.13\text{ nm}}\left(\frac{1}{2^2}-\frac{1}{n^2}\right) \quad\text{with }n=3,4,5,6,...$$
He predicted there would be even more spectral lines in the
hydrogen spectrum according to this Rydberg formula (1888).
$$\frac{1}{\lambda}=\frac{1}{91.13\text{ nm}}\left(\frac{1}{m^2}-\frac{1}{n^2}\right) \quad\text{with }m,n=1,2,3,4,5,...$$
And indeed,
soon experimental physicists found these series of spectral lines
in the ultraviolet and infrared part of the hydrogen spectrum.
$$\begin{align}
\text{Lyman series:}\quad    & \frac{1}{\lambda}=\frac{1}{91.13\text{ nm}}\left(\frac{1}{1^2}-\frac{1}{n^2}\right) & \text{with }n=2,3,4,5,... \\
\text{Paschen series:}\quad  & \frac{1}{\lambda}=\frac{1}{91.13\text{ nm}}\left(\frac{1}{3^2}-\frac{1}{n^2}\right) & \text{with }n=4,5,6,7,... \\
\text{Brackett series:}\quad & \frac{1}{\lambda}=\frac{1}{91.13\text{ nm}}\left(\frac{1}{4^2}-\frac{1}{n^2}\right) & \text{with }n=5,6,7,8,...
\end{align}$$
Again, there was no physical theory available yet.
This had to wait until the invention of quantum mechanics,
beginning with the Bohr model (1913) and its explanation
of the Rydberg formula.
