title: MotionLM: Multi-Agent Motion Forecasting as Language Modeling
summary: Reliable forecasting of the future behavior of road agents is a critical
  component to safe planning in autonomous vehicles. Here, we represent
  continuous trajectories as sequences of discrete motion tokens and cast
  multi-agent motion prediction as a language modeling task over this domain. Our
  model, MotionLM, provides several advantages: First, it does not require
  anchors or explicit latent variable optimization to learn multimodal
  distributions. Instead, we leverage a single standard language modeling
  objective, maximizing the average log probability over sequence tokens. Second,
  our approach bypasses post-hoc interaction heuristics where individual agent
  trajectory generation is conducted prior to interactive scoring. Instead,
  MotionLM produces joint distributions over interactive agent futures in a
  single autoregressive decoding process. In addition, the model's sequential
  factorization enables temporally causal conditional rollouts. The proposed
  approach establishes new state-of-the-art performance for multi-agent motion
  prediction on the Waymo Open Motion Dataset, ranking 1st on the interactive
  challenge leaderboard.
paper_authors._id: 65162e608fe117b68e33934a
paper_authors.name: Ari Seff
paper_authors._id: 65162e608fe117b68e33934b
paper_authors.name: Brian Cera
paper_authors._id: 65162e608fe117b68e33934c
paper_authors.name: Dian Chen
paper_authors._id: 65162e608fe117b68e33934d
paper_authors.name: Mason Ng
paper_authors._id: 65162e608fe117b68e33934e
paper_authors.name: Aurick Zhou
paper_authors._id: 65162e608fe117b68e33934f
paper_authors.name: Nigamaa Nayakanti
paper_authors._id: 65162e608fe117b68e339350
paper_authors.name: Khaled S. Refaat
paper_authors._id: 65162e608fe117b68e339351
paper_authors.name: Rami Al-Rfou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-02T06:59:48.552Z
paper_authors._id: 6449a9cfeb7db8f70fb7f798
paper_authors.avatarUrl: /avatars/a4a5a96e9c857f44d43456579cddc51a.svg
paper_authors.fullname: Rami Al-Rfou
paper_authors.type: user
paper_authors.user: rmyeid
paper_authors._id: 65162e608fe117b68e339352
paper_authors.name: Benjamin Sapp
paper_title: MotionLM: Multi-Agent Motion Forecasting as Language Modeling
paper_summary: Reliable forecasting of the future behavior of road agents is a critical
  component to safe planning in autonomous vehicles. Here, we represent
  continuous trajectories as sequences of discrete motion tokens and cast
  multi-agent motion prediction as a language modeling task over this domain. Our
  model, MotionLM, provides several advantages: First, it does not require
  anchors or explicit latent variable optimization to learn multimodal
  distributions. Instead, we leverage a single standard language modeling
  objective, maximizing the average log probability over sequence tokens. Second,
  our approach bypasses post-hoc interaction heuristics where individual agent
  trajectory generation is conducted prior to interactive scoring. Instead,
  MotionLM produces joint distributions over interactive agent futures in a
  single autoregressive decoding process. In addition, the model's sequential
  factorization enables temporally causal conditional rollouts. The proposed
  approach establishes new state-of-the-art performance for multi-agent motion
  prediction on the Waymo Open Motion Dataset, ranking 1st on the interactive
  challenge leaderboard.
paper_ai_keywords: multi-agent motion prediction language modeling motion tokens multimodal distributions autoregressive decoding process temporal causality conditional rollouts Waymo Open Motion Dataset interactive challenge leaderboard