Contrasting Multiple Representations with the Multi-Marginal Matching Gap
<|reference_start|>Contrasting Multiple Representations with the Multi-Marginal Matching Gap: Learning meaningful representations of complex objects that can be seen through multiple ($k\geq 3$) views or modalities is a core task in machine learning. Existing methods use losses originally intended for paired views, and extend them to $k$ views, either by instantiating $\tfrac12k(k-1)$ loss-pairs, or by using reduced embeddings, following a \textit{one vs. average-of-rest} strategy. We propose the multi-marginal matching gap (M3G), a loss that borrows tools from multi-marginal optimal transport (MM-OT) theory to simultaneously incorporate all $k$ views. Given a batch of $n$ points, each seen as a $k$-tuple of views subsequently transformed into $k$ embeddings, our loss contrasts the cost of matching these $n$ ground-truth $k$-tuples with the MM-OT polymatching cost, which seeks $n$ optimally arranged $k$-tuples chosen within these $n\times k$ vectors. While the exponential complexity $O(n^k$) of the MM-OT problem may seem daunting, we show in experiments that a suitable generalization of the Sinkhorn algorithm for that problem can scale to, e.g., $k=3\sim 6$ views using mini-batches of size $64~\sim128$. Our experiments demonstrate improved performance over multiview extensions of pairwise losses, for both self-supervised and multimodal tasks.<|reference_end|>
@article{piran2024contrasting,
    title={Contrasting Multiple Representations with the Multi-Marginal Matching
  Gap},
    author={Zoe Piran, Michal Klein, James Thornton, Marco Cuturi},
    journal={arXiv preprint arXiv:2405.19532},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.19532},
    primaryClass={cs.LG}
}