Why does basicity go down along the group 15 hydrides? As we go down Group 15, the sizes of atoms increase and these are the central atoms in their corresponding hydrides. Shouldn't the large size of central atom (in case of $\ce{Bi}$) actually help the lone pair inside the $\ce{sp^3}$ hybrid orbital to easily be donated and thus increase the basic character down the group?

$\ce{N}$ is small and shouldn't it hold lone pair with much more strength than others, but $\ce{NH3}$ seems to be strong base while $\ce{BiH3}$ is the weakest one.
The very idea of $\ce{sp^3}$ hybridization you are referring to is pretty much not there, starting from $\ce{PH3}$. Look at its bond angles, they are almost exactly $90^\circ$. This means that phosphorus contributes to the $\ce{P-H}$ bonds almost exclusively with its $\ce p$ orbitals, and the lone pair just sits there hidden on $\ce s$ orbital, rather than stick out as $\ce{sp^3}$.

The same is even more true for the following hydrides.

Also, with nitrogen being the most electronegative of all, its partial negative charge in $\ce{NH3}$ attracts $\ce{H+}$, which is not the case for $\ce{PH3}$ and the rest.