Splines feature selection 
*

*I remember reading that for a certain kind of splines, all basis functions have to be included when using feature selection methods such as stepwise regression or shrinkage.
The argument for that was that without including all basis functions the splines could not be constructed, hence, selecting only parts of the basis function would be wrong.


*A similar argument might hold for selecting parts of one-hot-encoding nominal features, although I cannot think for a good reason why one should not be able to just "throw" certain levels "away".
Now I'm looking for a reference to read up on this again.

For #2, I can think of two arguments. Unfortunately, they contradict each other.

*

*The feature is the entire factor variable, and coding the way we do is just a technicality for doing the math.


*You can throw out levels. If your factor is dog/cat/alligator, maybe we can regard dogs and cats as the same and think of the animals as mammal/reptile, and removing the variable helps model performance by lowering the variance (fewer parameters in the model) with only a small hit to bias, as dogs and cats are very similar.
I don’t have articulate rationale right now, but I can kind of see analogous arguments for the splines.
I tend to side with the first argument and even posted something along those lines last year.
However, feature selection is notoriously unstable, so all of this is a bit dubious.
Your bounty mentions wanting a reputable source. Try Frank Harrell’s “Why R?” talk or his Regression Modeling Strategies book for discussions about feature selection stability.
