Visually Consistent Hierarchical Image Classification
Seulki Park Youren Zhang Stella X. Yu Sara Beery Jonathan Huang
Hierarchical classification requires predicting an entire taxonomy, demanding both per-level accuracy and cross-level consistency. However, per-level accuracy often suffers because different levels rely on distinct visual cues, making it a multi-task problem.  For example, identifying specific bird species (e.g., Anna's hummingbird) depends on localized features like head and neck color, while distinguishing a bird from a plant requires global features like feathers versus leaves. Existing methods address this by using separate model components for each level, but this often leads to inconsistencies, as classifiers focus on unrelated regions.  

Our key insight is that classification across levels should be grounded in consistent visual cues that link fine details to broader structures. Instead of treating fine and coarse classification separately, our model first captures key details (e.g., beak, wings) for species recognition, then integrates them into global features (e.g., bird shape) for higher-level classification. For this, we introduce a hierarchical model that aligns fine-to-coarse semantic parsing with consistent visual segmentation. Additionally, we propose a tree-path KL divergence loss to enforce semantic consistency across levels.  Our approach significantly outperforms zero-shot CLIP and other state-of-the-art methods on hierarchical classification benchmarks.
Hierarchical Classification, Visual grounding
We highlight the significance of visual consistency in hierarchical classification that classifiers across levels should be grounded in consistent visual cues.