title: Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for
    Text-to-Image Generation
summary: Recent works demonstrate that using reinforcement learning (RL) with quality
  rewards can enhance the quality of generated images in text-to-image (T2I)
  generation. However, a simple aggregation of multiple rewards may cause
  over-optimization in certain metrics and degradation in others, and it is
  challenging to manually find the optimal weights. An effective strategy to
  jointly optimize multiple rewards in RL for T2I generation is highly desirable.
  This paper introduces Parrot, a novel multi-reward RL framework for T2I
  generation. Through the use of the batch-wise Pareto optimal selection, Parrot
  automatically identifies the optimal trade-off among different rewards during
  the RL optimization of the T2I generation. Additionally, Parrot employs a joint
  optimization approach for the T2I model and the prompt expansion network,
  facilitating the generation of quality-aware text prompts, thus further
  enhancing the final image quality. To counteract the potential catastrophic
  forgetting of the original user prompt due to prompt expansion, we introduce
  original prompt centered guidance at inference time, ensuring that the
  generated image remains faithful to the user input. Extensive experiments and a
  user study demonstrate that Parrot outperforms several baseline methods across
  various quality criteria, including aesthetics, human preference, image
  sentiment, and text-image alignment.
paper_authors._id: 65a0ad5990ea9d75eaa753b8
paper_authors.name: Seung Hyun Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:35:17.506Z
paper_authors._id: 6474b8c15aba8edfb2ef65aa
paper_authors.avatarUrl: /avatars/3331316476ef9f79cd4294e28c88cfe0.svg
paper_authors.fullname: Seung Hyun Lee
paper_authors.type: user
paper_authors.user: Seanlee235
paper_authors._id: 65a0ad5990ea9d75eaa753b9
paper_authors.name: Yinxiao Li
paper_authors._id: 65a0ad5990ea9d75eaa753ba
paper_authors.name: Junjie Ke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:36:21.063Z
paper_authors._id: 64736b148b7a55cfa91e000f
paper_authors.avatarUrl: /avatars/dcd2a666aba505575a79ffcc08a98c9d.svg
paper_authors.fullname: Ian Kerr
paper_authors.type: user
paper_authors.user: KeJunjie
paper_authors._id: 65a0ad5990ea9d75eaa753bb
paper_authors.name: Innfarn Yoo
paper_authors._id: 65a0ad5990ea9d75eaa753bc
paper_authors.name: Han Zhang
paper_authors._id: 65a0ad5990ea9d75eaa753bd
paper_authors.name: Jiahui Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:36:52.645Z
paper_authors._id: 6174394801833e378dfc2517
paper_authors.avatarUrl: /avatars/4989bcdda7ba1ae51cab8235c3e39275.svg
paper_authors.fullname: Jiahui Yu
paper_authors.type: user
paper_authors.user: jiahuiyu
paper_authors._id: 65a0ad5990ea9d75eaa753be
paper_authors.name: Qifei Wang
paper_authors._id: 65a0ad5990ea9d75eaa753bf
paper_authors.name: Fei Deng
paper_authors._id: 65a0ad5990ea9d75eaa753c0
paper_authors.name: Glenn Entis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:37:49.208Z
paper_authors._id: 6317bf4033a4e2b5d3b0aaaa
paper_authors.avatarUrl: /avatars/8a8b586e86d79520c5f0a866ed2f0b80.svg
paper_authors.fullname: Glenn Entis
paper_authors.type: user
paper_authors.user: GEinSF
paper_authors._id: 65a0ad5990ea9d75eaa753c1
paper_authors.name: Junfeng He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:37:55.773Z
paper_authors._id: 658224bc083a345666a037bc
paper_authors.avatarUrl: /avatars/a6719ec5f0a323b2504794dbf26a94d9.svg
paper_authors.fullname: Junfeng He
paper_authors.type: user
paper_authors.user: JF1980
paper_authors._id: 65a0ad5990ea9d75eaa753c2
paper_authors.name: Gang Li
paper_authors._id: 65a0ad5990ea9d75eaa753c3
paper_authors.name: Sangpil Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T09:38:11.190Z
paper_authors._id: 64fec50f6f77556d68d21fac
paper_authors.avatarUrl: /avatars/89dc61984a1fc7f8a09624186fe9fd53.svg
paper_authors.fullname: sangpil
paper_authors.type: user
paper_authors.user: kimsangpil
paper_authors._id: 65a0ad5990ea9d75eaa753c4
paper_authors.name: Irfan Essa
paper_authors._id: 65a0ad5990ea9d75eaa753c5
paper_authors.name: Feng Yang
paper_title: Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for
    Text-to-Image Generation
paper_summary: Recent works demonstrate that using reinforcement learning (RL) with quality
  rewards can enhance the quality of generated images in text-to-image (T2I)
  generation. However, a simple aggregation of multiple rewards may cause
  over-optimization in certain metrics and degradation in others, and it is
  challenging to manually find the optimal weights. An effective strategy to
  jointly optimize multiple rewards in RL for T2I generation is highly desirable.
  This paper introduces Parrot, a novel multi-reward RL framework for T2I
  generation. Through the use of the batch-wise Pareto optimal selection, Parrot
  automatically identifies the optimal trade-off among different rewards during
  the RL optimization of the T2I generation. Additionally, Parrot employs a joint
  optimization approach for the T2I model and the prompt expansion network,
  facilitating the generation of quality-aware text prompts, thus further
  enhancing the final image quality. To counteract the potential catastrophic
  forgetting of the original user prompt due to prompt expansion, we introduce
  original prompt centered guidance at inference time, ensuring that the
  generated image remains faithful to the user input. Extensive experiments and a
  user study demonstrate that Parrot outperforms several baseline methods across
  various quality criteria, including aesthetics, human preference, image
  sentiment, and text-image alignment.
paper_ai_summary: Parrot, a multi-reward RL framework, enhances text-to-image generation by automatically balancing multiple quality rewards and generating quality-aware text prompts without forgetting the original user input.
paper_ai_keywords: reinforcement learning quality rewards text-to-image generation batch-wise Pareto optimal selection prompt expansion network catastrophic forgetting original prompt centered guidance aesthetics human preference image sentiment text-image alignment