How can sodium be reduced in the Castner–Kellner process? In the Castner–Kellner process, why is sodium amalgam formed instead of hydrogen gas at the cathode?

If we electrolyse brine solution, then it will liberate hydrogen at cathode, not sodium, because $\ce{Na+}$ has a negative reduction potential while that of $\ce{H+}$ is (by definition) 0. What factor here causes the sodium to get reduced to form an amalgam in spite of the reduction potentials?
I am not sure if you are familiar with the term "overpotential." With mercury electrode, as a cathode, it takes more negative potential to reduce hydrogen ions to hydrogen gas, rather it is easier for sodium ions to get reduced on a mercury surface and form an amalgam (an alloy of Hg and Na). This is a peculiarity of the mercury electrode.

Mercury cell in chlor-alkali industry