Q:

Categorical predictors where response is 0/1 [decision trees]

In section 9.2.4 of Elements of Statistical Learning (pasted below for reference), it states that if the response variable is binary, then predictor classes can be ordered according to proportion falling in outcome class 1.
What does this mean? Is it saying that rather than splitting this categorical variable based on its categories, we split it based on the outcome of the class?
For example, consider a predictor $X_1$ the following values [cat, dog, bird, whale, dolphin, fish] and the response is [1, 0], where 1 represents mammal and 0 represents not mammal.
When we use $X_1$ to perform a split at a node, we split based on whether the examples are mammal and not mammal rather than, say, whether the example is a cat or not a cat, a dog or not a dog, etc...? I don't think this is what it's saying because that would be weird, but it sounds like it.

A:

It means you create an encoded variable whose values are based on the proportion of the binary classes instead of iterating over all possible combinations of the classes to find the best split. For example, in the following table:

A would be encoded as 1, B would be encoded as 2, C would be encoded as 3. This is much less expensive than determining how to break up A, B, and C into 2 groups.

