Hiformer: Heterogeneous Feature Interactions Learning with Transformers
  for Recommender Systems
Learning feature interaction is the critical backbone to building recommender
systems. In web-scale applications, learning feature interaction is extremely
challenging due to the sparse and large input feature space; meanwhile,
manually crafting effective feature interactions is infeasible because of the
exponential solution space. We propose to leverage a Transformer-based
architecture with attention layers to automatically capture feature
interactions. Transformer architectures have witnessed great success in many
domains, such as natural language processing and computer vision. However,
there has not been much adoption of Transformer architecture for feature
interaction modeling in industry. We aim at closing the gap. We identify two
key challenges for applying the vanilla Transformer architecture to web-scale
recommender systems: (1) Transformer architecture fails to capture the
heterogeneous feature interactions in the self-attention layer; (2) The serving
latency of Transformer architecture might be too high to be deployed in
web-scale recommender systems. We first propose a heterogeneous self-attention
layer, which is a simple yet effective modification to the self-attention layer
in Transformer, to take into account the heterogeneity of feature interactions.
We then introduce Hiformer (Heterogeneous
Interaction Transformer) to further improve the model
expressiveness. With low-rank approximation and model pruning, \hiformer enjoys
fast inference for online deployment. Extensive offline experiment results
corroborates the effectiveness and efficiency of the Hiformer model.
We have successfully deployed the Hiformer model to a real world large
scale App ranking model at Google Play, with significant improvement in key
engagement metrics (up to +2.66\%).
6551a00d0c11ee1eb90178ba
Huan Gui
6551a00d0c11ee1eb90178bb
Ruoxi Wang
admin_assigned
2023-11-13T09:30:52.120Z
62fafe5aefd2de854932e11c
/avatars/bd8fb1a958853e1e615574aa050c9086.svg
RuoxiWang
user
ruoxiwang
6551a00d0c11ee1eb90178bc
Ke Yin
6551a00d0c11ee1eb90178bd
Long Jin
extracted_confirmed
2023-11-14T02:17:31.510Z
6552d7f7d3b9b5be9383c352
/avatars/f39665ecacde7dd4cce681f21202562f.svg
Long Jin
user
longjin9
6551a00d0c11ee1eb90178be
Maciej Kula
6551a00d0c11ee1eb90178bf
Taibai Xu
6551a00d0c11ee1eb90178c0
Lichan Hong
6551a00d0c11ee1eb90178c1
Ed H. Chi
Hiformer: Heterogeneous Feature Interactions Learning with Transformers
  for Recommender Systems
Learning feature interaction is the critical backbone to building recommender
systems. In web-scale applications, learning feature interaction is extremely
challenging due to the sparse and large input feature space; meanwhile,
manually crafting effective feature interactions is infeasible because of the
exponential solution space. We propose to leverage a Transformer-based
architecture with attention layers to automatically capture feature
interactions. Transformer architectures have witnessed great success in many
domains, such as natural language processing and computer vision. However,
there has not been much adoption of Transformer architecture for feature
interaction modeling in industry. We aim at closing the gap. We identify two
key challenges for applying the vanilla Transformer architecture to web-scale
recommender systems: (1) Transformer architecture fails to capture the
heterogeneous feature interactions in the self-attention layer; (2) The serving
latency of Transformer architecture might be too high to be deployed in
web-scale recommender systems. We first propose a heterogeneous self-attention
layer, which is a simple yet effective modification to the self-attention layer
in Transformer, to take into account the heterogeneity of feature interactions.
We then introduce Hiformer (Heterogeneous
Interaction Transformer) to further improve the model
expressiveness. With low-rank approximation and model pruning, \hiformer enjoys
fast inference for online deployment. Extensive offline experiment results
corroborates the effectiveness and efficiency of the Hiformer model.
We have successfully deployed the Hiformer model to a real world large
scale App ranking model at Google Play, with significant improvement in key
engagement metrics (up to +2.66\%).
A novel Transformer-based architecture, Hiformer, addresses challenges in feature interaction for web-scale recommender systems by introducing a heterogeneous self-attention layer and optimizing for fast inference.
Transformer self-attention layer heterogeneous feature interactions Hiformer low-rank approximation model pruning web-scale recommender systems