title: ChatAnything: Facetime Chat with LLM-Enhanced Personas
summary: In this technical report, we target generating anthropomorphized personas for
  LLM-based characters in an online manner, including visual appearance,
  personality and tones, with only text descriptions. To achieve this, we first
  leverage the in-context learning capability of LLMs for personality generation
  by carefully designing a set of system prompts. We then propose two novel
  concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for
  diverse voice and appearance generation. For MoV, we utilize the text-to-speech
  (TTS) algorithms with a variety of pre-defined tones and select the most
  matching one based on the user-provided text description automatically. For
  MoD, we combine the recent popular text-to-image generation techniques and
  talking head algorithms to streamline the process of generating talking
  objects. We termed the whole framework as ChatAnything. With it, users could be
  able to animate anything with any personas that are anthropomorphic using just
  a few text inputs. However, we have observed that the anthropomorphic objects
  produced by current generative models are often undetectable by pre-trained
  face landmark detectors, leading to failure of the face motion generation, even
  if these faces possess human-like appearances because those images are nearly
  seen during the training (e.g., OOD samples). To address this issue, we
  incorporate pixel-level guidance to infuse human face landmarks during the
  image generation phase. To benchmark these metrics, we have built an evaluation
  dataset. Based on it, we verify that the detection rate of the face landmark is
  significantly increased from 57.0% to 92.5% thus allowing automatic face
  animation based on generated speech content. The code and more results can be
  found at https://chatanything.github.io/.
paper_authors._id: 6552f3bb0166ff6bd22f4628
paper_authors.name: Yilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:04:59.786Z
paper_authors._id: 6385d86181fe8c678a345ff0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6385d86181fe8c678a345ff0/ETvVOtBAQ-RpwfT4TCWsv.jpeg
paper_authors.fullname: Yilin Zhao
paper_authors.type: user
paper_authors.user: ermu2001
paper_authors._id: 6552f3bb0166ff6bd22f4629
paper_authors.name: Xinbin Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:02:25.005Z
paper_authors._id: 6512e44292a52e002ab78b6c
paper_authors.avatarUrl: /avatars/16bc674177346d19e012c1f27dc8495a.svg
paper_authors.fullname: Yuanxinbin
paper_authors.type: user
paper_authors.user: xinbinLL
paper_authors._id: 6552f3bb0166ff6bd22f462a
paper_authors.name: Shanghua Gao
paper_authors._id: 6552f3bb0166ff6bd22f462b
paper_authors.name: Zhijie Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:04:01.298Z
paper_authors._id: 64415957bd0c9726529802f6
paper_authors.avatarUrl: /avatars/1132d1ee68fb58ec635d57c8175caacd.svg
paper_authors.fullname: Zhijie Lin
paper_authors.type: user
paper_authors.user: Ikuinen
paper_authors._id: 6552f3bb0166ff6bd22f462c
paper_authors.name: Qibin Hou
paper_authors._id: 6552f3bb0166ff6bd22f462d
paper_authors.name: Jiashi Feng
paper_authors._id: 6552f3bb0166ff6bd22f462e
paper_authors.name: Daquan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:04:31.330Z
paper_authors._id: 63fe0b160c1bbe8e29d2dd32
paper_authors.avatarUrl: /avatars/bc574036287170a77057893efaa48e2d.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: DaQuan21
paper_title: ChatAnything: Facetime Chat with LLM-Enhanced Personas
paper_summary: In this technical report, we target generating anthropomorphized personas for
  LLM-based characters in an online manner, including visual appearance,
  personality and tones, with only text descriptions. To achieve this, we first
  leverage the in-context learning capability of LLMs for personality generation
  by carefully designing a set of system prompts. We then propose two novel
  concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for
  diverse voice and appearance generation. For MoV, we utilize the text-to-speech
  (TTS) algorithms with a variety of pre-defined tones and select the most
  matching one based on the user-provided text description automatically. For
  MoD, we combine the recent popular text-to-image generation techniques and
  talking head algorithms to streamline the process of generating talking
  objects. We termed the whole framework as ChatAnything. With it, users could be
  able to animate anything with any personas that are anthropomorphic using just
  a few text inputs. However, we have observed that the anthropomorphic objects
  produced by current generative models are often undetectable by pre-trained
  face landmark detectors, leading to failure of the face motion generation, even
  if these faces possess human-like appearances because those images are nearly
  seen during the training (e.g., OOD samples). To address this issue, we
  incorporate pixel-level guidance to infuse human face landmarks during the
  image generation phase. To benchmark these metrics, we have built an evaluation
  dataset. Based on it, we verify that the detection rate of the face landmark is
  significantly increased from 57.0% to 92.5% thus allowing automatic face
  animation based on generated speech content. The code and more results can be
  found at https://chatanything.github.io/.
paper_ai_summary: A framework for generating anthropomorphized personas with diverse voices and appearances from text descriptions using LLMs and generative models, with improved face landmark detection for automatic animation.
paper_ai_keywords: LLM-based characters in-context learning mixture of voices mixture of diffusers text-to-speech text-to-image talking head algorithms ChatAnything pixel-level guidance face landmark detection out-of-distribution (OOD) samples