Why do we need so many dummy variables in a regression with categorical predictor? Why not use binary encoding instead of one-hot encoding? If we have $k$ categories of a categorical variable, why do we need $k-1$ dummy variables?
For example, if there are 8 categories, why don't we code them as
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1?

Only 3 dummy variables would be needed as opposed to the commonly used $8-1 = 7$ dummy variables.

For one, the predictor variables would not be orthogonal. Generally, linear regressions work better (i.e., have tighter error estimates) when predictors are close to orthogonal (which can be quantified using the condition number of the Gram matrix $X^tX$). But the bigger issue with this proposed coding scheme is that it introduces similarity structure to the features that is completely arbitrary. For example the first row and the second row are less similar than the first row and the third row (as measured by, e.g., dot product similarity), but in a categorical variable the labeling of the values (or what is the same thing, the ordering of the rows of the encoding matrix) has no inherent meaning. So combining these two considerations, the only way that really makes sense to code a categorical variable with $K$ values is to choose $k$ orthogonal vectors, which requires a space of dimension $>=K$ (or really, $>=K-1$ since you can absorb one of them into the intercept). The typical coding using one-hotes is a particularly simple way to accomplish this, although in principle you could also use any set of $K$ orthogonal vectors.
