Wukong: Towards a Scaling Law for Large-Scale Recommendation
Scaling laws play an instrumental role in the sustainable improvement in
model quality. Unfortunately, recommendation models to date do not exhibit such
laws similar to those observed in the domain of large language models, due to
the inefficiencies of their upscaling mechanisms. This limitation poses
significant challenges in adapting these models to increasingly more complex
real-world datasets. In this paper, we propose an effective network
architecture based purely on stacked factorization machines, and a synergistic
upscaling strategy, collectively dubbed Wukong, to establish a scaling law in
the domain of recommendation. Wukong's unique design makes it possible to
capture diverse, any-order of interactions simply through taller and wider
layers. We conducted extensive evaluations on six public datasets, and our
results demonstrate that Wukong consistently outperforms state-of-the-art
models quality-wise. Further, we assessed Wukong's scalability on an internal,
large-scale dataset. The results show that Wukong retains its superiority in
quality over state-of-the-art models, while holding the scaling law across two
orders of magnitude in model complexity, extending beyond 100 Gflop or
equivalently up to GPT-3/LLaMa-2 scale of total training compute, where prior
arts fall short.
65e7d73e3fb61ba9c613dd89
Buyun Zhang
65e7d73e3fb61ba9c613dd8a
Liang Luo
claimed_verified
2024-03-07T09:15:23.487Z
65e8be4b45ae4e903cee573a
/avatars/546e503dc13a1c081c6b139f551b5b00.svg
Liang Luo
user
liangluo
65e7d73e3fb61ba9c613dd8b
Yuxin Chen
65e7d73e3fb61ba9c613dd8c
Jade Nie
65e7d73e3fb61ba9c613dd8d
Xi Liu
65e7d73e3fb61ba9c613dd8e
Daifeng Guo
65e7d73e3fb61ba9c613dd8f
Yanli Zhao
65e7d73e3fb61ba9c613dd90
Shen Li
65e7d73e3fb61ba9c613dd91
Yuchen Hao
65e7d73e3fb61ba9c613dd92
Yantao Yao
65e7d73e3fb61ba9c613dd93
Guna Lakshminarayanan
65e7d73e3fb61ba9c613dd94
Ellie Dingqiao Wen
65e7d73e3fb61ba9c613dd95
Jongsoo Park
admin_assigned
2024-03-06T10:37:32.025Z
64470b86cffecebe8b5fb8e4
/avatars/89e60436c278f028ec96968a1230472e.svg
Jongsoo Park
user
jongsoopark
65e7d73e3fb61ba9c613dd96
Maxim Naumov
65e7d73e3fb61ba9c613dd97
Wenlin Chen
Wukong: Towards a Scaling Law for Large-Scale Recommendation
Scaling laws play an instrumental role in the sustainable improvement in
model quality. Unfortunately, recommendation models to date do not exhibit such
laws similar to those observed in the domain of large language models, due to
the inefficiencies of their upscaling mechanisms. This limitation poses
significant challenges in adapting these models to increasingly more complex
real-world datasets. In this paper, we propose an effective network
architecture based purely on stacked factorization machines, and a synergistic
upscaling strategy, collectively dubbed Wukong, to establish a scaling law in
the domain of recommendation. Wukong's unique design makes it possible to
capture diverse, any-order of interactions simply through taller and wider
layers. We conducted extensive evaluations on six public datasets, and our
results demonstrate that Wukong consistently outperforms state-of-the-art
models quality-wise. Further, we assessed Wukong's scalability on an internal,
large-scale dataset. The results show that Wukong retains its superiority in
quality over state-of-the-art models, while holding the scaling law across two
orders of magnitude in model complexity, extending beyond 100 Gflop or
equivalently up to GPT-3/LLaMa-2 scale of total training compute, where prior
arts fall short.
Wukong, a network architecture using stacked factorization machines with a synergistic upscaling strategy, establishes a scaling law in recommendation systems, outperforming state-of-the-art models and maintaining quality across significant increases in complexity.
scaling laws factorization machines upscaling strategy network architecture interactions quality scalability large-scale datasets