title: Hiformer: Heterogeneous Feature Interactions Learning with Transformers
    for Recommender Systems
summary: Learning feature interaction is the critical backbone to building recommender
  systems. In web-scale applications, learning feature interaction is extremely
  challenging due to the sparse and large input feature space; meanwhile,
  manually crafting effective feature interactions is infeasible because of the
  exponential solution space. We propose to leverage a Transformer-based
  architecture with attention layers to automatically capture feature
  interactions. Transformer architectures have witnessed great success in many
  domains, such as natural language processing and computer vision. However,
  there has not been much adoption of Transformer architecture for feature
  interaction modeling in industry. We aim at closing the gap. We identify two
  key challenges for applying the vanilla Transformer architecture to web-scale
  recommender systems: (1) Transformer architecture fails to capture the
  heterogeneous feature interactions in the self-attention layer; (2) The serving
  latency of Transformer architecture might be too high to be deployed in
  web-scale recommender systems. We first propose a heterogeneous self-attention
  layer, which is a simple yet effective modification to the self-attention layer
  in Transformer, to take into account the heterogeneity of feature interactions.
  We then introduce Hiformer (Heterogeneous
  Interaction Transformer) to further improve the model
  expressiveness. With low-rank approximation and model pruning, \hiformer enjoys
  fast inference for online deployment. Extensive offline experiment results
  corroborates the effectiveness and efficiency of the Hiformer model.
  We have successfully deployed the Hiformer model to a real world large
  scale App ranking model at Google Play, with significant improvement in key
  engagement metrics (up to +2.66\%).
paper_authors._id: 6551a00d0c11ee1eb90178ba
paper_authors.name: Huan Gui
paper_authors._id: 6551a00d0c11ee1eb90178bb
paper_authors.name: Ruoxi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T09:30:52.120Z
paper_authors._id: 62fafe5aefd2de854932e11c
paper_authors.avatarUrl: /avatars/bd8fb1a958853e1e615574aa050c9086.svg
paper_authors.fullname: RuoxiWang
paper_authors.type: user
paper_authors.user: ruoxiwang
paper_authors._id: 6551a00d0c11ee1eb90178bc
paper_authors.name: Ke Yin
paper_authors._id: 6551a00d0c11ee1eb90178bd
paper_authors.name: Long Jin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-14T02:17:31.510Z
paper_authors._id: 6552d7f7d3b9b5be9383c352
paper_authors.avatarUrl: /avatars/f39665ecacde7dd4cce681f21202562f.svg
paper_authors.fullname: Long Jin
paper_authors.type: user
paper_authors.user: longjin9
paper_authors._id: 6551a00d0c11ee1eb90178be
paper_authors.name: Maciej Kula
paper_authors._id: 6551a00d0c11ee1eb90178bf
paper_authors.name: Taibai Xu
paper_authors._id: 6551a00d0c11ee1eb90178c0
paper_authors.name: Lichan Hong
paper_authors._id: 6551a00d0c11ee1eb90178c1
paper_authors.name: Ed H. Chi
paper_title: Hiformer: Heterogeneous Feature Interactions Learning with Transformers
    for Recommender Systems
paper_summary: Learning feature interaction is the critical backbone to building recommender
  systems. In web-scale applications, learning feature interaction is extremely
  challenging due to the sparse and large input feature space; meanwhile,
  manually crafting effective feature interactions is infeasible because of the
  exponential solution space. We propose to leverage a Transformer-based
  architecture with attention layers to automatically capture feature
  interactions. Transformer architectures have witnessed great success in many
  domains, such as natural language processing and computer vision. However,
  there has not been much adoption of Transformer architecture for feature
  interaction modeling in industry. We aim at closing the gap. We identify two
  key challenges for applying the vanilla Transformer architecture to web-scale
  recommender systems: (1) Transformer architecture fails to capture the
  heterogeneous feature interactions in the self-attention layer; (2) The serving
  latency of Transformer architecture might be too high to be deployed in
  web-scale recommender systems. We first propose a heterogeneous self-attention
  layer, which is a simple yet effective modification to the self-attention layer
  in Transformer, to take into account the heterogeneity of feature interactions.
  We then introduce Hiformer (Heterogeneous
  Interaction Transformer) to further improve the model
  expressiveness. With low-rank approximation and model pruning, \hiformer enjoys
  fast inference for online deployment. Extensive offline experiment results
  corroborates the effectiveness and efficiency of the Hiformer model.
  We have successfully deployed the Hiformer model to a real world large
  scale App ranking model at Google Play, with significant improvement in key
  engagement metrics (up to +2.66\%).
paper_ai_summary: A novel Transformer-based architecture, Hiformer, addresses challenges in feature interaction for web-scale recommender systems by introducing a heterogeneous self-attention layer and optimizing for fast inference.
paper_ai_keywords: Transformer self-attention layer heterogeneous feature interactions Hiformer low-rank approximation model pruning web-scale recommender systems