&gt;&gt; He&#x27;ll happily throw in your zip code and credit card number into the mix, not realizing that those two features have super high entropy<p>You will be surprised: you can make significant gains by including the zip code - i&#x27;ve seen that happen in a competitive setting. Where you live probably contains some signal about your credit worthiness.<p>Having said that, of course it doesn&#x27;t make sense to simply feed the raw zip code to the tree. An appropriate encoding (most people would use a one-hot encoding, though there exist better ones) of the zip code will be key to extracting signal in a robust way.<p>&gt;&gt; ... the tree will have serious overfitting issues<p>Isn&#x27;t it an almost standard practise now to use an ensemble of trees, such as a random forest? Decision trees have long been known to be prone to overfitting.