title: Cognitively Inspired Energy-Based World Models
summary: One of the predominant methods for training world models is autoregressive
  prediction in the output space of the next element of a sequence. In Natural
  Language Processing (NLP), this takes the form of Large Language Models (LLMs)
  predicting the next token; in Computer Vision (CV), this takes the form of
  autoregressive models predicting the next frame/token/pixel. However, this
  approach differs from human cognition in several respects. First, human
  predictions about the future actively influence internal cognitive processes.
  Second, humans naturally evaluate the plausibility of predictions regarding
  future states. Based on this capability, and third, by assessing when
  predictions are sufficient, humans allocate a dynamic amount of time to make a
  prediction. This adaptive process is analogous to System 2 thinking in
  psychology. All these capabilities are fundamental to the success of humans at
  high-level reasoning and planning. Therefore, to address the limitations of
  traditional autoregressive models lacking these human-like capabilities, we
  introduce Energy-Based World Models (EBWM). EBWM involves training an
  Energy-Based Model (EBM) to predict the compatibility of a given context and a
  predicted future state. In doing so, EBWM enables models to achieve all three
  facets of human cognition described. Moreover, we developed a variant of the
  traditional autoregressive transformer tailored for Energy-Based models, termed
  the Energy-Based Transformer (EBT). Our results demonstrate that EBWM scales
  better with data and GPU Hours than traditional autoregressive transformers in
  CV, and that EBWM offers promising early scaling in NLP. Consequently, this
  approach offers an exciting path toward training future models capable of
  System 2 thinking and intelligently searching across state spaces.
paper_authors._id: 666ba21cd3dbd0378e786601
paper_authors.name: Alexi Gladstone
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-14T01:59:28.888Z
paper_authors._id: 63fbbff41b4b1bd4e7042fd1
paper_authors.avatarUrl: /avatars/d186d0859046b35723fe1156d8f68917.svg
paper_authors.fullname: Alexi Gladstone
paper_authors.type: user
paper_authors.user: alexiglad
paper_authors._id: 666ba21cd3dbd0378e786602
paper_authors.name: Ganesh Nanduru
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:55:39.526Z
paper_authors._id: 64f518ecdf78ce3cc8526639
paper_authors.avatarUrl: /avatars/7b6ac7b1ff061f392fccf4af66f831d2.svg
paper_authors.fullname: Ganesh Nanduru
paper_authors.type: user
paper_authors.user: gnanduru
paper_authors._id: 666ba21cd3dbd0378e786603
paper_authors.name: Md Mofijul Islam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:55:45.553Z
paper_authors._id: 60d599155a14347cd1f6c6eb
paper_authors.avatarUrl: /avatars/c623abb79f6dc48b54f2fb8ba64bb4a4.svg
paper_authors.fullname: Md Mofijul Islam
paper_authors.type: user
paper_authors.user: mmiakashs
paper_authors._id: 666ba21cd3dbd0378e786604
paper_authors.name: Aman Chadha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:55:50.871Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 666ba21cd3dbd0378e786605
paper_authors.name: Jundong Li
paper_authors._id: 666ba21cd3dbd0378e786606
paper_authors.name: Tariq Iqbal
paper_title: Cognitively Inspired Energy-Based World Models
paper_summary: One of the predominant methods for training world models is autoregressive
  prediction in the output space of the next element of a sequence. In Natural
  Language Processing (NLP), this takes the form of Large Language Models (LLMs)
  predicting the next token; in Computer Vision (CV), this takes the form of
  autoregressive models predicting the next frame/token/pixel. However, this
  approach differs from human cognition in several respects. First, human
  predictions about the future actively influence internal cognitive processes.
  Second, humans naturally evaluate the plausibility of predictions regarding
  future states. Based on this capability, and third, by assessing when
  predictions are sufficient, humans allocate a dynamic amount of time to make a
  prediction. This adaptive process is analogous to System 2 thinking in
  psychology. All these capabilities are fundamental to the success of humans at
  high-level reasoning and planning. Therefore, to address the limitations of
  traditional autoregressive models lacking these human-like capabilities, we
  introduce Energy-Based World Models (EBWM). EBWM involves training an
  Energy-Based Model (EBM) to predict the compatibility of a given context and a
  predicted future state. In doing so, EBWM enables models to achieve all three
  facets of human cognition described. Moreover, we developed a variant of the
  traditional autoregressive transformer tailored for Energy-Based models, termed
  the Energy-Based Transformer (EBT). Our results demonstrate that EBWM scales
  better with data and GPU Hours than traditional autoregressive transformers in
  CV, and that EBWM offers promising early scaling in NLP. Consequently, this
  approach offers an exciting path toward training future models capable of
  System 2 thinking and intelligently searching across state spaces.
paper_ai_keywords: autoregressive prediction Large Language Models (LLMs) token Computer Vision (CV) autoregressive models frame pixel Energy-Based World Models (EBWM) Energy-Based Model (EBM) Energy-Based Transformer (EBT) System 2 thinking traditional autoregressive transformers