Lowering the weight of particular features in a neural network? Given sample data $x$, we hypothesize that some features (i.e. dimensions) of $x$ will generalize well, while others will generalize poorly.  For example, when predicting medical diagnosis, age and blood pressure will likely generalize much better than zip code, last name, or patient-id.  It is common to drop these features, especially if they have a large domain set size (e.g. there may be many last names, with only a few samples per each name).
However, being forced to drop them entirely is unfortunate, because these features may have important signals.  E.g. zip codes may show environmental effects, and last names may capture family history.  Ideally, we'd like to be able to use these "poorer features" as part of the input, but discount their effect on the output.  For example, if the "good features" produce a high confidence prediction, ignore the "poor features", but if the good features produce an ambivalent prediction, use the "poor features".
Assume we are using a neural network.  One way to achieve this is to scale down the input features by their presumed quality.  E.g. Instead of representing a categorical feature by a one-hot, represent it by a "0.1-hot".  Therefore, this input will have less impact on the final prediction, but still available.

*

*Is this a good approach?

*Are there other approaches?

*How do we prevent the model from learning to rescale the signal back up? Especially given the affine transformation of batch normalization, scaling things down may not have much impact at all.


Bayesian approaches allow you to impart on the model your outside domain knowledge, such as zip code mattering but probably only a little bit. A nice part about this is that you really do have domain knowledge that you want to give your model.
Two drawbacks of Bayesian approaches are that a strong prior can be hard to overcome, and there can be lengthy computations.
Nonetheless, you might have some luck looking into Bayesian regression and neural networks.
