title: LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction
    Tuning
summary: Generating instructional images of human daily actions from an egocentric
  viewpoint serves a key step towards efficient skill transfer. In this paper, we
  introduce a novel problem -- egocentric action frame generation. The goal is to
  synthesize the action frame conditioning on the user prompt question and an
  input egocentric image that captures user's environment. Notably, existing
  egocentric datasets lack the detailed annotations that describe the execution
  of actions. Additionally, the diffusion-based image manipulation models fail to
  control the state change of an action within the corresponding egocentric image
  pixel space. To this end, we finetune a visual large language model (VLLM) via
  visual instruction tuning for curating the enriched action descriptions to
  address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO)
  action frame generation using image and text embeddings from VLLM as additional
  conditioning. We validate our proposed model on two egocentric datasets --
  Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior
  image manipulation models in both quantitative and qualitative evaluation. We
  also conduct detailed ablation studies and analysis to provide insights on our
  method.
paper_authors._id: 65727c32e887face3eb87572
paper_authors.name: Bolin Lai
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-08T18:56:59.365Z
paper_authors._id: 64dbcdaa70446182be84353e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64dbcdaa70446182be84353e/wjzSeLKOdg3krOlkn24ea.jpeg
paper_authors.fullname: Bolin Lai
paper_authors.type: user
paper_authors.user: bolinlai
paper_authors._id: 65727c32e887face3eb87573
paper_authors.name: Xiaoliang Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:03:52.545Z
paper_authors._id: 6549417b3ce45eb764faf993
paper_authors.avatarUrl: /avatars/d310f475d0697f5f13b3d4141ea0ccaf.svg
paper_authors.fullname: Xiaoliang Dai
paper_authors.type: user
paper_authors.user: daixl1992
paper_authors._id: 65727c32e887face3eb87574
paper_authors.name: Lawrence Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T11:05:21.721Z
paper_authors._id: 6303ff0c7b50dd9d0a3777e8
paper_authors.avatarUrl: /avatars/1c65d5936fe0cafd1fadd1102fad28db.svg
paper_authors.fullname: Lawrence Chen
paper_authors.type: user
paper_authors.user: lawrencechen
paper_authors._id: 65727c32e887face3eb87575
paper_authors.name: Guan Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-11T08:16:41.294Z
paper_authors._id: 6573a458d7f487de5f9bb44a
paper_authors.avatarUrl: /avatars/4aa5626af43855ad770186f0d4191a01.svg
paper_authors.fullname: Guan Pang
paper_authors.type: user
paper_authors.user: tbpangolin
paper_authors._id: 65727c32e887face3eb87576
paper_authors.name: James M. Rehg
paper_authors._id: 65727c32e887face3eb87577
paper_authors.name: Miao Liu
paper_title: LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction
    Tuning
paper_summary: Generating instructional images of human daily actions from an egocentric
  viewpoint serves a key step towards efficient skill transfer. In this paper, we
  introduce a novel problem -- egocentric action frame generation. The goal is to
  synthesize the action frame conditioning on the user prompt question and an
  input egocentric image that captures user's environment. Notably, existing
  egocentric datasets lack the detailed annotations that describe the execution
  of actions. Additionally, the diffusion-based image manipulation models fail to
  control the state change of an action within the corresponding egocentric image
  pixel space. To this end, we finetune a visual large language model (VLLM) via
  visual instruction tuning for curating the enriched action descriptions to
  address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO)
  action frame generation using image and text embeddings from VLLM as additional
  conditioning. We validate our proposed model on two egocentric datasets --
  Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior
  image manipulation models in both quantitative and qualitative evaluation. We
  also conduct detailed ablation studies and analysis to provide insights on our
  method.
paper_ai_summary: A model for generating egocentric action frames from user prompts and images using a fine-tuned visual large language model and learned embeddings improves upon existing image manipulation models.
paper_ai_keywords: visual large language model visual instruction tuning egocentric action frame generation action frame synthesis enriched action descriptions image embeddings text embeddings Ego4D Epic-Kitchens