ARIG: Autoregressive Interactive Head Generation for Real-time
  Conversations
Face-to-face communication, as a common human activity, motivates the
research on interactive head generation. A virtual agent can generate motion
responses with both listening and speaking capabilities based on the audio or
motion signals of the other user and itself. However, previous clip-wise
generation paradigm or explicit listener/speaker generator-switching methods
have limitations in future signal acquisition, contextual behavioral
understanding, and switching smoothness, making it challenging to be real-time
and realistic. In this paper, we propose an autoregressive (AR) based
frame-wise framework called ARIG to realize the real-time generation with
better interaction realism. To achieve real-time generation, we model motion
prediction as a non-vector-quantized AR process. Unlike discrete codebook-index
prediction, we represent motion distribution using diffusion procedure,
achieving more accurate predictions in continuous space. To improve interaction
realism, we emphasize interactive behavior understanding (IBU) and detailed
conversational state understanding (CSU). In IBU, based on dual-track
dual-modal signals, we summarize short-range behaviors through
bidirectional-integrated learning and perform contextual understanding over
long ranges. In CSU, we use voice activity signals and context features of IBU
to understand the various states (interruption, feedback, pause, etc.) that
exist in actual conversations. These serve as conditions for the final
progressive motion prediction. Extensive experiments have verified the
effectiveness of our model.
68669614c415dc4bb9b2fd81
Ying Guo
68669614c415dc4bb9b2fd82
Xi Liu
68669614c415dc4bb9b2fd83
Cheng Zhen
68669614c415dc4bb9b2fd84
Pengfei Yan
68669614c415dc4bb9b2fd85
Xiaoming Wei
ARIG: Autoregressive Interactive Head Generation for Real-time
  Conversations
Face-to-face communication, as a common human activity, motivates the
research on interactive head generation. A virtual agent can generate motion
responses with both listening and speaking capabilities based on the audio or
motion signals of the other user and itself. However, previous clip-wise
generation paradigm or explicit listener/speaker generator-switching methods
have limitations in future signal acquisition, contextual behavioral
understanding, and switching smoothness, making it challenging to be real-time
and realistic. In this paper, we propose an autoregressive (AR) based
frame-wise framework called ARIG to realize the real-time generation with
better interaction realism. To achieve real-time generation, we model motion
prediction as a non-vector-quantized AR process. Unlike discrete codebook-index
prediction, we represent motion distribution using diffusion procedure,
achieving more accurate predictions in continuous space. To improve interaction
realism, we emphasize interactive behavior understanding (IBU) and detailed
conversational state understanding (CSU). In IBU, based on dual-track
dual-modal signals, we summarize short-range behaviors through
bidirectional-integrated learning and perform contextual understanding over
long ranges. In CSU, we use voice activity signals and context features of IBU
to understand the various states (interruption, feedback, pause, etc.) that
exist in actual conversations. These serve as conditions for the final
progressive motion prediction. Extensive experiments have verified the
effectiveness of our model.
An autoregressive framework for real-time interactive head generation improves interaction realism through bidirectional learning and detailed conversational state understanding.
autoregressive frame-wise motion prediction diffusion procedure bidirectional-integrated learning interactive behavior understanding conversational state understanding progressive motion prediction