Bootstrapping World Models from Dynamics Models in Multimodal Foundation
  Models
To what extent do vision-and-language foundation models possess a realistic
world model (observation times action rightarrow observation) and a
dynamics model (observation times observation rightarrow action), when
actions are expressed through language? While open-source foundation models
struggle with both, we find that fine-tuning them to acquire a dynamics model
through supervision is significantly easier than acquiring a world model. In
turn, dynamics models can be used to bootstrap world models through two main
strategies: 1) weakly supervised learning from synthetic data and 2) inference
time verification. Firstly, the dynamics model can annotate actions for
unlabelled pairs of video frame observations to expand the training data. We
further propose a new objective, where image tokens in observation pairs are
weighted by their importance, as predicted by a recognition model. Secondly,
the dynamics models can assign rewards to multiple samples of the world model
to score them, effectively guiding search at inference time. We evaluate the
world models resulting from both strategies through the task of action-centric
image editing on Aurora-Bench. Our best model achieves a performance
competitive with state-of-the-art image editing models, improving on them by a
margin of 15% on real-world subsets according to GPT4o-as-judge, and
achieving the best average human evaluation across all subsets of Aurora-Bench.
68480800308cb7e626e80e20
Yifu Qiu
claimed_verified
2025-06-10T10:59:54.843Z
64686434f43574d9556b1fa6
/avatars/64183d643cfc3b274714a6167c354e39.svg
Yifu Qiu
user
yfqiu-nlp
68480800308cb7e626e80e21
Yftah Ziser
68480800308cb7e626e80e22
Anna Korhonen
68480800308cb7e626e80e23
Shay B. Cohen
68480800308cb7e626e80e24
Edoardo M. Ponti
claimed_verified
2025-06-10T10:59:52.211Z
60809ad44ad99100d63ce36a
https://cdn-avatars.huggingface.co/v1/production/uploads/1619040921084-noauth.jpeg
Edoardo Maria Ponti
user
ducdauge
Bootstrapping World Models from Dynamics Models in Multimodal Foundation
  Models
To what extent do vision-and-language foundation models possess a realistic
world model (observation times action rightarrow observation) and a
dynamics model (observation times observation rightarrow action), when
actions are expressed through language? While open-source foundation models
struggle with both, we find that fine-tuning them to acquire a dynamics model
through supervision is significantly easier than acquiring a world model. In
turn, dynamics models can be used to bootstrap world models through two main
strategies: 1) weakly supervised learning from synthetic data and 2) inference
time verification. Firstly, the dynamics model can annotate actions for
unlabelled pairs of video frame observations to expand the training data. We
further propose a new objective, where image tokens in observation pairs are
weighted by their importance, as predicted by a recognition model. Secondly,
the dynamics models can assign rewards to multiple samples of the world model
to score them, effectively guiding search at inference time. We evaluate the
world models resulting from both strategies through the task of action-centric
image editing on Aurora-Bench. Our best model achieves a performance
competitive with state-of-the-art image editing models, improving on them by a
margin of 15% on real-world subsets according to GPT4o-as-judge, and
achieving the best average human evaluation across all subsets of Aurora-Bench.
Foundation models can be fine-tuned to develop dynamics models more easily than world models, with dynamics models aiding world model development through weak supervision and inference time verification, leading to state-of-the-art performance in action-centric image editing.
vision-and-language foundation models realistic world model dynamics model fine-tuning weakly supervised learning synthetic data inference time verification image tokens recognition model action-centric image editing Aurora-Bench GPT4o-as-judge