Is it important to convert numeric features to object if they have no ordinal or mathematical meaning? For example a column containing numeric values for phone area code or a postal code.
In case it matters, I am preprocessing data for use in a tree-based ensemble classifier.

Yes, it is important, otherwise, a model cannot correctly learn effects for individual factors. However, some tree-based implementations do not do that because it is computationally more efficient to have one variable with 100 numeric values than to have 100 dummy variables, and a tree will eventually split all levels anyway, so the arbitrary ordering is not such a big issue as it is in other models, like linear regression.
If there is information "accidentally" encoded in the numeric code, then I would argue that you should still treat them as categorical variables and instead put that information explicitly in your model. So if a zip code is roughly related to geographic location, then it would be better to put the categorical zipcode in location to your model explicitly than treating zipcode as a numeric feature and hoping a model will figure it out. Also, if there is some information encoded in the numerical values, you should be extra cautious because this might lead to data leakage problems
