title: Wukong: Towards a Scaling Law for Large-Scale Recommendation
summary: Scaling laws play an instrumental role in the sustainable improvement in
  model quality. Unfortunately, recommendation models to date do not exhibit such
  laws similar to those observed in the domain of large language models, due to
  the inefficiencies of their upscaling mechanisms. This limitation poses
  significant challenges in adapting these models to increasingly more complex
  real-world datasets. In this paper, we propose an effective network
  architecture based purely on stacked factorization machines, and a synergistic
  upscaling strategy, collectively dubbed Wukong, to establish a scaling law in
  the domain of recommendation. Wukong's unique design makes it possible to
  capture diverse, any-order of interactions simply through taller and wider
  layers. We conducted extensive evaluations on six public datasets, and our
  results demonstrate that Wukong consistently outperforms state-of-the-art
  models quality-wise. Further, we assessed Wukong's scalability on an internal,
  large-scale dataset. The results show that Wukong retains its superiority in
  quality over state-of-the-art models, while holding the scaling law across two
  orders of magnitude in model complexity, extending beyond 100 Gflop or
  equivalently up to GPT-3/LLaMa-2 scale of total training compute, where prior
  arts fall short.
paper_authors._id: 65e7d73e3fb61ba9c613dd89
paper_authors.name: Buyun Zhang
paper_authors._id: 65e7d73e3fb61ba9c613dd8a
paper_authors.name: Liang Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:15:23.487Z
paper_authors._id: 65e8be4b45ae4e903cee573a
paper_authors.avatarUrl: /avatars/546e503dc13a1c081c6b139f551b5b00.svg
paper_authors.fullname: Liang Luo
paper_authors.type: user
paper_authors.user: liangluo
paper_authors._id: 65e7d73e3fb61ba9c613dd8b
paper_authors.name: Yuxin Chen
paper_authors._id: 65e7d73e3fb61ba9c613dd8c
paper_authors.name: Jade Nie
paper_authors._id: 65e7d73e3fb61ba9c613dd8d
paper_authors.name: Xi Liu
paper_authors._id: 65e7d73e3fb61ba9c613dd8e
paper_authors.name: Daifeng Guo
paper_authors._id: 65e7d73e3fb61ba9c613dd8f
paper_authors.name: Yanli Zhao
paper_authors._id: 65e7d73e3fb61ba9c613dd90
paper_authors.name: Shen Li
paper_authors._id: 65e7d73e3fb61ba9c613dd91
paper_authors.name: Yuchen Hao
paper_authors._id: 65e7d73e3fb61ba9c613dd92
paper_authors.name: Yantao Yao
paper_authors._id: 65e7d73e3fb61ba9c613dd93
paper_authors.name: Guna Lakshminarayanan
paper_authors._id: 65e7d73e3fb61ba9c613dd94
paper_authors.name: Ellie Dingqiao Wen
paper_authors._id: 65e7d73e3fb61ba9c613dd95
paper_authors.name: Jongsoo Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:37:32.025Z
paper_authors._id: 64470b86cffecebe8b5fb8e4
paper_authors.avatarUrl: /avatars/89e60436c278f028ec96968a1230472e.svg
paper_authors.fullname: Jongsoo Park
paper_authors.type: user
paper_authors.user: jongsoopark
paper_authors._id: 65e7d73e3fb61ba9c613dd96
paper_authors.name: Maxim Naumov
paper_authors._id: 65e7d73e3fb61ba9c613dd97
paper_authors.name: Wenlin Chen
paper_title: Wukong: Towards a Scaling Law for Large-Scale Recommendation
paper_summary: Scaling laws play an instrumental role in the sustainable improvement in
  model quality. Unfortunately, recommendation models to date do not exhibit such
  laws similar to those observed in the domain of large language models, due to
  the inefficiencies of their upscaling mechanisms. This limitation poses
  significant challenges in adapting these models to increasingly more complex
  real-world datasets. In this paper, we propose an effective network
  architecture based purely on stacked factorization machines, and a synergistic
  upscaling strategy, collectively dubbed Wukong, to establish a scaling law in
  the domain of recommendation. Wukong's unique design makes it possible to
  capture diverse, any-order of interactions simply through taller and wider
  layers. We conducted extensive evaluations on six public datasets, and our
  results demonstrate that Wukong consistently outperforms state-of-the-art
  models quality-wise. Further, we assessed Wukong's scalability on an internal,
  large-scale dataset. The results show that Wukong retains its superiority in
  quality over state-of-the-art models, while holding the scaling law across two
  orders of magnitude in model complexity, extending beyond 100 Gflop or
  equivalently up to GPT-3/LLaMa-2 scale of total training compute, where prior
  arts fall short.
paper_ai_keywords: factorization machines scaling laws upscaling mechanisms large language models neural networks interactions layers scalability large-scale dataset Gflop GPT-3 LLaMa-2