Emergent Interpretable Symbols and Content-Style Disentanglement via Variance-Invariance Constraints
<|reference_start|>Emergent Interpretable Symbols and Content-Style Disentanglement via Variance-Invariance Constraints: We contribute an unsupervised method that effectively learns from raw observation and disentangles its latent space into content and style representations. Unlike most disentanglement algorithms that rely on domain-specific labels and knowledge, our method is based on the insight of domain-general statistical differences between content and style -- content varies more among different fragments within a sample but maintains an invariant vocabulary across data samples, whereas style remains relatively invariant within a sample but exhibits more significant variation across different samples. We integrate such inductive bias into an encoder-decoder architecture and name our method after V3 (variance-versus-invariance). Experimental results show that V3 generalizes across two distinct domains in different modalities, music audio and images of written digits, successfully learning pitch-timbre and digit-color disentanglements, respectively. Also, the disentanglement robustness significantly outperforms baseline unsupervised methods and is even comparable to supervised counterparts. Furthermore, symbolic-level interpretability emerges in the learned codebook of content, forging a near one-to-one alignment between machine representation and human knowledge.<|reference_end|>
@article{wu2024emergent,
    title={Emergent Interpretable Symbols and Content-Style Disentanglement via
  Variance-Invariance Constraints},
    author={Yuxuan Wu, Ziyu Wang, Bhiksha Raj, Gus Xia},
    journal={arXiv preprint arXiv:2407.03824},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.03824},
    primaryClass={cs.LG cs.AI}
}