Splines feature selection 
*

*I remember reading that for a certain kind of splines, all basis functions have to be included when using feature selection methods such as stepwise regression or shrinkage.
The argument for that was that without including all basis functions the splines could not be constructed, hence, selecting only parts of the basis function would be wrong.


*A similar argument might hold for selecting parts of one-hot-encoding nominal features, although I cannot think for a good reason why one should not be able to just "throw" certain levels "away".
Now I'm looking for a reference to read up on this again.

Just as it is not appropriate to not let $x$ and $x^2$ terms be dropped in a quadratic regression, it is important to treat all basis functions as a block.  If the sample size doesn't support estimating all the parameters involved, then use penalization/shrinkage/regularization to effectively give the block of basis functions fewer degrees of freedom then are apparent.
Also I'd say that dropping levels of a categorical predictor is problematic when using supervised learning (i.e., when utilizing $Y$). This will result in biased (low) standard errors and poor frequentist operating characteristics.  Like nonlinear basis functions it's better to penalize all the indicator variables involved in a multi-level categorical predictor than to drop some of the indicators.
