Teaching Metric Distance to Autoregressive Multimodal Foundational
  Models
As large language models expand beyond natural language to domains such as
mathematics, multimodal understanding, and embodied agents, tokens increasingly
reflect metric relationships rather than purely linguistic meaning. We
introduce DIST2Loss, a distance-aware framework designed to train
autoregressive discrete models by leveraging predefined distance relationships
among output tokens. At its core, DIST2Loss transforms continuous exponential
family distributions derived from inherent distance metrics into discrete,
categorical optimization targets compatible with the models' architectures.
This approach enables the models to learn and preserve meaningful distance
relationships during token generation while maintaining compatibility with
existing architectures. Empirical evaluations show consistent performance gains
in diverse multimodal applications, including visual grounding, robotic
manipulation, generative reward modeling, and image generation using
vector-quantized features. These improvements are pronounced in cases of
limited training data, highlighting DIST2Loss's effectiveness in
resource-constrained settings.
67c7acfb3c08da184dd8463e
Jiwan Chung
claimed_verified
2025-06-03T08:55:37.936Z
60d74d1affe0328e0167dc5f
/avatars/9b1a2df9402e9c26e1eb7c818af9bae0.svg
Jiwan Chung
user
jiwan-chung
67c7acfb3c08da184dd8463f
Saejin Kim
67c7acfb3c08da184dd84640
Yongrae Jo
67c7acfb3c08da184dd84641
Jaewoo Park
67c7acfb3c08da184dd84642
Dongjun Min
67c7acfb3c08da184dd84643
Youngjae Yu
Teaching Metric Distance to Autoregressive Multimodal Foundational
  Models
As large language models expand beyond natural language to domains such as
mathematics, multimodal understanding, and embodied agents, tokens increasingly
reflect metric relationships rather than purely linguistic meaning. We
introduce DIST2Loss, a distance-aware framework designed to train
autoregressive discrete models by leveraging predefined distance relationships
among output tokens. At its core, DIST2Loss transforms continuous exponential
family distributions derived from inherent distance metrics into discrete,
categorical optimization targets compatible with the models' architectures.
This approach enables the models to learn and preserve meaningful distance
relationships during token generation while maintaining compatibility with
existing architectures. Empirical evaluations show consistent performance gains
in diverse multimodal applications, including visual grounding, robotic
manipulation, generative reward modeling, and image generation using
vector-quantized features. These improvements are pronounced in cases of
limited training data, highlighting DIST2Loss's effectiveness in
resource-constrained settings.
DIST2Loss, a distance-aware framework, enhances the performance of autoregressive models by incorporating distance relationships among tokens, improving multimodal applications with limited data.
DIST2Loss autoregressive discrete models distance-aware framework exponential family distributions categorical optimization targets token generation visual grounding robotic manipulation generative reward modeling image generation vector-quantized features