## Semantic Codebook Learning for Dynamic Recommendation Models

Zheqi Lv Zhejiang University Hangzhou, China zheqilv@zju.edu.cn

Shengyu Zhang ∗ Zhejiang University Hangzhou, China Shanghai Institute for Advanced Study, Zhejiang University Shanghai, China sy\_zhang@zju.edu.cn

## Shaoxuan He

Zhejiang University Hangzhou, China shxhe@zju.edu.cn

Tianyu Zhan Zhejiang University Hangzhou, China yuzt@zju.edu.cn

Wenqiao Zhang Zhejiang University Hangzhou, China wenqiaozhang@zju.edu.cn

Jingyuan Chen Zhejiang University Hangzhou, China jingyuanchen@zju.edu.cn

Zhou Zhao ∗

Zhejiang University Hangzhou, China zhaozhou@zju.edu.cn

## ABSTRACT

Dynamic sequential recommendation (DSR) can generate model parameters based on user behavior to improve the personalization of sequential recommendation under various user preferences. However, it faces the challenges of large parameter search space and sparse and noisy user-item interactions, which reduces the applicability of the generated model parameters. The Semantic Codebook Learning for Dynamic Recommendation Models (SOLID) framework presents a significant advancement in DSR by effectively tackling these challenges. By transforming item sequences into semantic sequences and employing a dual parameter model, SOLID compresses the parameter generation search space and leverages homogeneity within the recommendation system. The introduction of the semantic metacode and semantic codebook, which stores disentangled item representations, ensures robust and accurate parameter generation. Extensive experiments demonstrates that SOLID consistently outperforms existing DSR, delivering more accurate, stable, and robust recommendations.

## CCS CONCEPTS

- · Information systems → Personalization ; Multimedia and multimodal retrieval .

∗ Corresponding authors.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

MM'24, October 28-November 1, 2024, Melbourne, VIC, Australia

© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 979-8-4007-0686-8/24/10

https://doi.org/10.1145/3664647.3680574

Fei Wu Zhejiang University

Hangzhou, China wufei@zju.edu.cn

## KEYWORDS

Semenatic Codebook, Dynamic Model, Disentangle, Sequential Recommendation, Multimodal, Personalization

## ACMReference Format:

Zheqi Lv, Shaoxuan He, Tianyu Zhan, Shengyu Zhang, Wenqiao Zhang, Jingyuan Chen, Zhou Zhao, and Fei Wu. 2024. Semantic Codebook Learning for Dynamic Recommendation Models. In Proceedings of the 32nd ACM International Conference on Multimedia (MM '24), October 28-November 1, 2024, Melbourne, VIC, AustraliaProceedings of the 32nd ACM International Conference on Multimedia (MM'24), October 28-November 1, 2024, Melbourne, Australia. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/ 3664647.3680574

## 1 INTRODUCTION

Nowadays, as an important branch of recommendation systems, sequential recommendation has emerged, including DIN [60], GRU4Rec [12], SASRec [17], BERT4Rec [34] and other models that are crucial in the field of recommendation systems. However, the behavior logic of most users is not universally applicable, and as interests can change, it necessitates that sequence recommendation models be able to adjust their parameters in real-time according to the user's current interest preferences. Consequently, dynamic sequential recommendation models (DSR) like DUET [29] and APG [48] have been developed.

The DSR paradigm consists of two parts: (1) The primary model. This model has a structure similar to conventional sequential recommendation models like SASRec, but it is divided into a static layer and a dynamic layer. The parameters of the static layer remain unchanged after pre-training, whereas the parameters of the dynamic layer change with the user's behavior. (2) The parameter generation model. This is mainly used to sparse user behavior and generate the parameters for the dynamic layer of the primary model based on this behavior. The DSR paradigm enables traditional static sequential recommendation models to quickly adjust

Figure 1: (a) describes multimodal user behavior data that includes images, text, and IDs. (b) describes the forward propagation of DSR, which is divided into two pathways: the first pathway processes user behavior data composed of IDs through a parameter generator to produce the parameters for the dynamic layers of the primary model. The second pathway processes the same ID-based user behavior data through the primary model's static layer, then through the dynamic layer, resulting in the prediction output. (c) and (d) compare the parameter generation patterns of existing DSR and SOLID. (e) compares the performance of our method and SR models and DSR Models on four multi modal recommendation datasets and four single modal recommendation datasets. The results show that our method significantly enhances performance on extensive datasets.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00123v1-figure-deed7f8b-4c95-498d-94f2-ba4043b09019.jpg)

their parameters according to the potential shift of interests and intentions reflected in user behaviors, thus dynamically obtaining more interest-aligned models in real time.

Despite the promising potential of Dynamic Sequential Recommendation (DSR) systems, they face significant challenges, primarily stemming from the item-to-parameter modeling scheme: (1) A large number of items result in a vast search space for the parameter generation model. Slight variations in user behavior sequences, such as "shirt, tie, suit" versus "tie, shirt, suit, " which suggest similar preferences, can unpredictably alter the item-to-parameter modeling, introducing complexity and potential instability. (2) The interaction between users and items is generally sparse and potentially noisy ( e.g. , the notorious implicit feedback issue), leading to heterogeneous behavior sequences that complicate the learning of accurate item representations. This results in inaccurate item representation learning, weakening the precision of model parameter customization based on item sequence features, and further exacerbating the inaccuracy of generated parameters.

behavior space modeling in the parameter generator. Moreover, SOLID shifts from a traditional item sequence-based parameter generation mode to a dual (item sequence + semantic sequence) → model parameter generation mode, effectively merging both uniform and diverse information in a structured manner. Uniform information derived from the semantic-to-parameter part is utilized to develop parameters that generalize across certain user behaviors, while diverse information allows for the crafting of specific parameters tailored to individual behavioral nuances. Crucially, by aligning the dimensions of the codebook with those of the semantic encoder, we transform the semantic encoder into a meta-code that serves as an initial state for the codebook, further easing the modeling of parameter generation.

To address these issues, we propose the S emantic C o debook L earning for Dynam c Recommen i d ation Models (SOLID). The core objective of SOLID is to compress the search space of the parameter generation model, promoting homogeneity signals utilization within the recommendation system. We construct a semantic codebook that better utilizes these homogeneity signals. In the codebook, item representations are disentangled into semantics that are learned to be absorbed in the codebook elements, such that the homogeneity between items in the disentangled latent space can be established. The user-item interactions are transformed into density-enriched user-semantic interactions in the latent space. The enriched density reduces the heterogeneity and complexity of user

Specifically, to reduce the search space of the parameter generation model through the semantic codebook, SOLID involves three main modules. Initially, SOLID employs a pretrained model to extract semantic components from item, image, and text features. This disentanglement transitions the focus from item sequences to semantic sequences, shifting the modeling approach from itembased to semantics-based parameter generation. This design results in trunk parameters that generalize behaviors from the entire user base to specific groups, and branch parameters that cater to individual user behaviors, both derived from semantic and item sequences respectively. Parameters derived from items are tightly controlled (e.g., ±0.01) before their integration into the dynamic layer of the primary model, ensuring a responsive and adaptive system based on real-time user activity. Despite this, branch parameters still adhere to an item-centric approach, necessitating the use of a Semantic Codebook (SC) to maintain personalization and stability in representation. This codebook stores semantic vectors of behavior,

progressively aligned with the nearest matches during learning. The weights of the semantic encoder are used to initialize the SC, easing the semantic codebook learning. As shown in Figure 1, SOLID is designed to pursue the precision, stability, and clarity of model parameter generation, trying to promote the dynamic recommendation model's response to sparse, heterogeneous, and potentially noisy user behaviors.

Our contributions can be summarized as:

- · We pointed the limitations of the existing DSR paradigm and designed the SOLID framework to address these deficiencies.
- · Wefirst learned to disentangle the parameter generation mode, which ensures that the generated model parameters contain both common and personalized knowledge.
- · We transformed the semantic encoder into a semantic metacode to enhance the semantic codebook learning.
- · We conducted extensive experiments on multiple datasets, which demonstrates the rationality and efficacy of SOLID.

## 2 RELATED WORK

## 2.1 Sequential Recommendation

Recommendation system predicts user preferences based on user behavior history [7, 19, 20, 22-25, 32, 33, 47, 51, 52, 56, 57]. Sequential recommendation, as an important branch of the recommendation system, arranges users' recent historical behaviors in chronological order to more accurately capture users' recent preferences. Recent advancements [4, 12, 17, 26, 27, 29, 34, 45, 48, 60] have shifted towards deep learning-based sequential recommendation systems. For instance, GRU4Rec [12] employs Gated Recurrent Units to effectively model sequential behavior, demonstrating impressive results. Additionally, DIN [60] and SASRec [17] incorporate attention mechanisms and transformers, respectively. BERT4Rec [34] further applies BERT for superior outcomes in recommendation task. The models have significantly impacted academic research and industry practices. However, these SR Models struggle to achieve optimal performance across every data distribution when dealing with users' real-time changing behaviors and interest preferences.

## 2.2 Disentangled Representation Learning

The goal of disentangled representation learning is to parse the data into distinct, interpretable components by identifying different underlying latent factors [2, 3]. Variational autoencoders (VAE) [5] and 𝛽 -VAE [13] provide more possibilities for disentangled learning by adjusting the balance between the model's disentanglement ability and its ability to represent information. By incorporating multiinterest methods [18, 30] along with disentangled representation learning, several studies [41-44, 58] have demonstrated significant advancements in recommendation tasks. We draw on the idea of disentangling and apply it to dynamic model parameter generation to reduce the parameter search space and leverage the homogeneous information of user behavior.

## 2.3 Dynamic Neural Network

Research in dynamic neural networks focuses on HyperNetworks [11] and Dynamic Filter Networks [16], which have better ability to adapt to distribution deviations than traditional static model learning or other efficient fine-tuning strategies [6, 9, 14, 15, 21, 36, 38, 39, 55, 59, 63, 65]. Similar situations also exist in the study of large models [53, 61, 62, 64]. HyperNetworks, introduced by Ha et al. [11], use one neural network to dynamically generate parameters for another, reducing the number of parameters needed and achieving model compression. This concept has led to extensive exploration and enhancements in various applications [1, 8, 10, 31, 35, 37, 46, 50, 53, 54]. Some recent research includes: HyperInverter [8], HyperStyle [1], Detective [54] introduces dynamic neural networks into multiple computer vision tasks to improve the model's personalization capabilities under various data distributions. IntellectReq [28] detects when such dynamic networks need to modify parameters to adapt to samples, thereby achieving better performance with fewer parameter modifier calls. APG[48] and DUET [29] are the latest and state-of-the-art examples of using dynamic neural networks for sequence recommendation. However, existing DSR models are affected by the heterogeneity of user behavior, the sparsity of user-item interactions, etc., leading to drawbacks such as an overly large parameter search space and inaccurate parameter generation. Our method effectively addresses these shortcomings.

## 3 METHODOLOGY

## 3.1 Notations and Problem Formulation

First, we introduce the notation in sequential recommendations.

3.1.1 Data. We use X ori = { 𝑢, 𝑣, 𝑠 𝑣 } to represent a piece of data, X dec = { 𝑢, 𝑐, 𝑠 𝑐 } to represent a piece of disentangled data, X mm = { 𝑖, 𝑡 } to represent multimodal information, and Y = { 𝑦 } to represent the label indicating whether the user will interact with the item. In brief, X = X ori ∪ X dec ∪ X mm = { 𝑢, 𝑣, 𝑠 𝑣 , 𝑐, 𝑠 𝑐 , 𝑖, 𝑡 } , where 𝑢, 𝑣, 𝑐, 𝑠 𝑣 , 𝑠 𝑐 , 𝑖, 𝑡 represent user ID, item ID, category ID, user's click sequence consists of item ID, user's click sequence consists of category ID, the image of the item, and the title of the item respectively. We represent the dataset as D , where D = { 𝑋,𝑌 } . More specifically, we use D Train to represent the training set and D Test to represent the test set. Roughly speaking, let L be the loss obtained from training on dataset D Train . For simplicity, we simplify the symbol D Train to D . Then, the model parameters 𝑊 can be obtained through the optimization function arg min L . The sequence length inputted into the model is set to 𝐿 𝑠 , so the lengths of both 𝑠 𝑣 and 𝑠 𝑐 in a sample are 𝐿 𝑠 .

3.1.2 Model. The recommendation model is represented by M and the parameters of the M is Θ , where Θ = Θ 𝑠 , Θ 𝑑 . The model M 𝑣 is utilized to generate the Θ 𝑑 according to the item id sequence 𝑠 𝑣 , M 𝑐 is utilized to generate the Θ 𝑑 according to the category id sequence 𝑠 𝑐 , M(·) and M (·) 𝑣 represent the forward propagation processes of two models, where · denotes the input.

3.1.3 Feature. We use Ev and Ec to represent the item feature set and semantic feature set extracted from 𝑠 𝑣 and 𝑠 𝑐 respectively. Specifically, Ev = { 𝑒 1 𝑣 , 𝑒 2 𝑣 , ..., 𝑒 𝐿 𝑠 𝑣 } , Ec = { 𝑒 1 𝑐 , 𝑒 2 𝑐 , ..., 𝑒 𝐿 𝑠 𝑐 } . ev and ec are the sequence features obtained through sequence feature extraction models such as Transformer or GRU, via Ev and Ec , respectively. The length of an item representation or a semantic representation is set to 𝐿 𝑟 .

Data

Inject

Figure 2: The framework of the SOLID, which consists of three main modules: Semantic Parameter Generation (SPG), Semantic Metacode Learning (SML), and Semantic Codebook Learning (SCL). SPG first converts item representations into semantics and constructs a semantic sequence to generate parameters in a structured manner. Subsequently, SML generates model parameters based on both the item sequence and the semantic sequence, and it jointly trains the model, accommodating both homogeneous and heterogeneous information. More importantly, the semantic encoder it learns can be transformed into metacode, which then provides a good initial value for the codebook. Finally, SCL learns a semantic codebook to improve the process of the parameter generation. Among them, L Rec = 𝑙 CE ( 𝑦,𝑦 , ˆ ) L Con = 𝑙 MSE ( E 𝑣 , E ′ 𝑣 ) .

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00123v1-figure-cc2d76a7-5e47-4c23-9dc0-0bc173201b46.jpg)

In the training stage, the loss can be calculated to optimize the sequential recommendation models as follows,

𝑒

Image Text Semantic Sequence 𝑒 ! " 𝑒 ! # 𝑒 $ % 𝑒 $ Semantic Metacode Learning Trunk 3.1.4 Formula. Sequential Recommendation Models (SR), Dynamic Sequential Recommendation Models (DSR), and Disentangled Multimodal Dynamic Sequential Recommendation Models (SOLID) can be formalized as follows:

$

ID

$$\min _ { \Theta } \mathcal { L } = \sum _ { u, v, s _ { v }, y \in \mathcal { D } } l _ { \text{CE} } ( y, \hat { y } = \mathcal { M } ( u, v, s _ { v } ; \Theta ) ). \quad ( 4 )$$

$$\text{SR} \colon \underbrace { \mathcal { M } ( \mathcal { X } _ { \text{ori} ; \Theta ) } } _ { \text{Recommendation Procedure } } \stackrel { \text{Gradients} } { \stackrel { \text{Gradients} } { \text{Output} } } \stackrel { \text{Gradients} } { \stackrel { \text{(\hat{Y} \Longleftrightarrow Y)} } { \text{Loss Calculation} } } } \text{ }. \quad ( 1 ) \quad \text{The lo}$$

$$\text{DSR } \colon \underbrace { \text{$M(X_{ori} ; \Theta_{s}, \Theta_{d} = M_{v}(X_{ori})$} ) } _ { \text{Recommendation Procedure} } \stackrel { \text{Gradients} } { \text{Output} } \quad \underbrace { \text{(\hat{Y} \Longleftrightarrow y ) } _ { \text{Loss Calculation} } } _ { \text{Loss Calculation} } \. \ ( 2 ) \quad \text{and} \ t i o n \, l$$

↦

$$\text{SOLID} \colon \begin{cases} \mathcal { X } _ { \text{ori} }, \mathcal { X } _ { \text{mm} } \mapsto c = f ( v, i, t ) \mapsto \mathcal { X } _ { \text{dec} }, & \text{the u page} \\ \Theta _ { d } = \mathcal { M } _ { v } ( \mathcal { X } _ { \text{ori} } ) \oplus \mathcal { M } _ { c } ( \mathcal { X } _ { \text{dec} } ), & \stackrel { \text{gradients} } { \underbrace { \text{Output} } } \stackrel { \text{gradients} } { \longleftarrow } & \underbrace { ( \hat { y } \Longleftarrow y ) } { \text{Loss Calculation} } \quad \text{$\mathfrak{$d$}$} \\ \text{in the aforementioned formula, $a\rightarrow b$ indicates indicates informa-} \quad \text{featu} \end{cases}$$

↦

In the aforementioned formula, 𝑎 → 𝑏 indicates indicates information transfer from 𝑎 to 𝑏 , with the text next to it representing the content of the transfer. 𝑎 → 𝑏 signifies that 𝑏 is derived from 𝑎 .

## 3.2 Preliminary

3.2.1 Sequential Recommendation Models. Here we first retrospect the paradigm of sequential recommendation.

𝑧 ' &amp; Transformation Layers The loss function can set to CE (Cross Entropy) loss and MSE (Mean Squared Error) loss, etc. However, since sequential recommendation often focuses more on CTR (Click-Through Rate) prediction tasks, and this paper is also focused on CTR prediction, the recommendation loss in this paper is CE loss and represented by 𝑙 CE.

𝑒

3.2.2 Dynamic Sequential Recommendation Models. DSR generate model parameters based on users' real-time user behaviors. Then the updated model is used for current recommendations. In this paper, the network layer that can adjust model parameters as the data distribution changes is called an adaptive layer.

DSR treat the parameters of one of the adaptive layers as a matrix 𝐾 ∈ R 𝑁 𝑖𝑛 × 𝑁 𝑜𝑢𝑡 , where 𝑁 𝑖𝑛 and 𝑁 𝑜𝑢𝑡 represent the number of input neurons and output neurons of a fully connected layer (FCL), respectively. DSR utilize a encoder 𝐸 𝑣 to extract the sequence feature 𝒆 𝑣 from the user's behavior sequence 𝑠 𝑣 to generate the parameters of the model's adaptive layers.

$$\theta _ { d } = \mathcal { M } _ { v } ( E _ { v } ( s _ { v } ) ),$$

After parameter generation, the parameters of the model will be reshaped into the shape of 𝐾 .

During training, all layers of the M 𝑣 are optimized together with the static layers of the M . The loss function L is defined as follows:

$$\min _ { \Theta _ { s }, \Theta _ { v } } \mathcal { L } = \sum _ { u, v, s _ { v }, y \in \mathcal { D } } l _ { \text{CE} } ( y, \hat { y } = \mathcal { M } ( u, v, s _ { v } ; \Theta _ { s }, \Theta _ { d } ) ). \quad ( 6 ) \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

Although the Item-based Dynamic Recommendation Model can obtain personalized model parameters based on users' real-time behavior and achieve superior performance, it also faces multiple challenges. 1) The user-item interaction is extremely sparse, leading to inaccurate item representation learning, making the model parameters customized based on item-based features inaccurate. 2) The personalized model parameters obtained by this strategy are highly mixed. 3) The generated parameters are not subject to any constraints, which poses challenges to the stability of the generated model. So we design the novel methods to address the challenges mentioned above.

## 3.3 SOLID Framework

The architecture of our proposed SOLID is shown in the Figure 2.

3.3.1 Semantic Parameter Generation. Transforming the Item-based Dynamic Recommendation Model into a Semantic-based Dynamic Recommendation Model is an important step in disentangling personalized model parameters. First, items need to be transformed into semantics. For data without category labels, clustering can be directly applied to obtain semantics, i.e.,

$$\text{Cluster} ( \{ e _ { i } \} _ { i = 1 } ^ { N } ) \mapsto \{ c _ { i } \} _ { i = 1 } ^ { N }, c _ { i } \in \{ 1, 2,.., k \}. \quad \text{(7)} \quad \text{be 1}$$

For data with category labels, since the same item often belongs to multiple categories, we select a primary category as semantic it. First, we define the centroid 𝑚 𝑐 of each category 𝑐 , which is the average of embeddings 𝑒 for all items belonging to category 𝑐 . Assuming 𝑛 𝑐 is the number of items belonging to category 𝑐 , the centroid 𝑚 𝑐 for category 𝑐 can be represented as:

$$m _ { c } = \frac { 1 } { n _ { c } } \sum _ { v \in c } ( e _ { v } \, \text{or} \, e _ { i } \, \text{or} \, e _ { t } ), \quad \quad ( 8 ) \quad \stackrel { \prime } { s o }.$$

where 𝑒 𝑣 , 𝑒 𝑖 , 𝑒 𝑡 are the representation of item ID 𝑣 , item image 𝑖 , item title 𝑡 , respectively. Next, we compute its distance to each category center 𝑚 𝑐 . Assuming we use the Euclidean distance, it can be represented as,

$$d ( v, c ) = \| ( e _ { v } \text{ or } e _ { i } \text{ or } e _ { t } ) - m _ { c } \|, \quad \quad ( 9 ) \quad \ g e$$

where ∥ · ∥ denotes the norm of the vector, typically the Euclidean norm. Finally, we select the closest category as the semantic for item 𝑣 . That is, the semantic 𝑐 𝑝 for item 𝑣 can be represented as:

$$c _ { p } = \arg \min _ { c } d ( ( v \, \text{or} \, i \, \text{or} \, t ), c ). \quad \quad ( 1 0 ) \quad \text{it.}$$

After converting items into semantics, a semantic-to-parameter model can be trained. The training process is similar to that of the item-to-parameter model. The only differences are that the input for the item-to-parameter model is an item sequence, whereas for the semantic-to-parameter model, it is a semantic sequence; similarly, the outputs are the target item and target semantic, respectively.

$$\Gamma \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \tan \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \ sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \ cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \osigma \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \exp \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \lesssim \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \colon = \sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \sin \sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \ sin \sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \exp \sin \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \os \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \loss \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \prod \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \ros \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \none \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \sec \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \yes \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \cos \$$



In the above equation, 𝐸 𝑐 represents the semantic encoder, which is similar to the item encoder 𝐸 𝑣 .

3.3.2 Semantic Metacode Learning. To balance the use of personalized user behavior information and homogeneous information from similar user behaviors, we combine the item-to-parameter and semantic-to-parameter models for the parameter generation process. The former's advantage lies in providing personalized information, but its disadvantage is the inaccuracy in parameter generation due to strong data heterogeneity and sparse user-item interactions. The latter's advantage is providing homogeneous information from similar user behaviors, and dense user-item interactions make the parameter generation process more robust. However, its disadvantage is that the semantic sequence is less personalized compared to the item sequence.

Therefore, our approach primarily uses the semantic-to-parameter method to generate the main part of the model parameters. Since similar semantic sequences are easier to obtain than similar item sequences, the parameters derived from the semantic sequence can be viewed as a user group model. Then, the item-to-parameter method is used as a branch, with parameters generated from item sequences being constrained within a smaller threshold and merged with the parameters obtained from the semantic sequence. This merging process is seen as a transition from a user group model to an individual user model, thus balancing homogeneous information and personalized information. Therefore, the training process can be formulated as the following optimization problem,

$$\dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \delta$$



where T is a hyperparameter used to control the threshold for parameter deviation, thereby also controlling the impact of personalized information on the model parameters. Semantic Encoder can be transformed into a Semantic Metacode(SM), which can be used to further enhance the initialization of the Semantic Codebook for the item-to-parameter process. The Semantic Metacode can be effectively learned through the above process.

3.3.3 Semantic Codebook Learning. Even if the model parameter generation process is disentangled, the item-to-parameter mode is still needed because it is the source of personalized information. Therefore, to further improve the accuracy of the item-to-parameter mapping, we design a Semantic Codebook (SC). Upon obtaining the semantic metacode, we initialize the semantic codebook with it. Subsequently, we continue using the trunk and branch method of parameter generation, specifically semantic-to-parameter and item-to-parameter, to derive the parameters for the adaptive layer of the model. In the branch branch, the item representations are replaced with semantic codes from the codebook, which are then used to further predict model parameters. The generated model parameters are used for click prediction on item sequences, just as before, ultimately allowing for the training of the semantic codebook. The specific method for computing the loss is described below. SC is denoted as 𝐷 , and 𝐷 ∈ R N × 𝑐 𝐿 𝑟 . Specifically, we first use the weights of the semantic encoder in the semantic-to-parameter to initialize the item representation, as their dimensions are the same.

Then, we encode the user's item representation. For a piece of data, as introduced in the notation description section, its item representation is E 𝑣 = { 𝑒 1 𝑣 , 𝑒 2 𝑣 , ..., 𝑒 𝐿 𝑠 𝑣 } . Afterward, we find the closest feature in the SC to replace each item representation in the set E 𝑣 , obtaining E ′ 𝑣 = { 𝑒 ′ 1 𝑣 , 𝑒 ′ 2 𝑣 , ..., 𝑒 ′ 𝐿 𝑠 𝑣 } , and the sequence feature obtained from E ′ 𝑣 is 𝑒 ′ 𝑣 . Subsequently, we compute the MSE loss between the item representation set E ′ 𝑣 obtained from the SC and the original set E 𝑣 , and incorporate it into the training process as follows,

$$\begin{cases} \underset { \Theta _ { s }, \Theta _ { c }, \Theta _ { v } } { \mu } & \mathcal { L } = \sum _ { u, v, s _ { c }, y \in \mathcal { D } } l _ { \text{CE} } ( y, \hat { y } ) + \lambda l _ { \text{MSE} } ( \text{E} _ { v }, \text{E} _ { v } ^ { \prime } ), & \quad \\ \hat { y } = \mathcal { M } ( u, v, s _ { v } ; \Theta _ { s }, \Theta _ { d } ), & ( 1 3 ) & \quad \\ \Theta _ { d } = \mathcal { M } _ { c } ( e _ { c } ) + \text{Clip} ( \mathcal { M } _ { v } ( e ^ { \prime } _ { v } ) ) ; \mathcal { T } ), & \quad \\ \cdot & \cdot & \cdot & \dots \cdot & \cdot & \cdot & \end{cases}$$



where 𝑙 MSE represents the MSE loss, and the 𝜆 is a hyperparameter.

3.3.4 Pseudo Code of SOLID. Algorithm 1 shows the pseudo code of SOLID. ( 𝑥 ) represents that 𝑥 is a intermediate variable.

## 4 EXPERIMENTS

## 4.1 Experimental Setup

4.1.1 Datasets and Preprocessing. We evaluate SOLID and baselines on eight datasets. Amazon Arts (Arts) , Amazon Instruments (Instruments) Amazon Office (Office) Amazon , , Scientific (Scientific) , which are four benchmarks that was recently released but has been widely used in the multimodal recommendation tasks [40]. Amazon CDs (CDs) Amazon Electronic , (Electronic) Douban Book (Book) , , and Douban Music (Music) , which are four widely used public benchmarks in the recommendation tasks. We choose the leave-one-out approach to process the dataset, taking the last action of each user for testing and all previous actions for training and validation. Our task is CTR (Clickthrough Rate) prediction, so we process these datasets into CTR prediction datasets. These datasets consist of user rating datasets with complete reviews. We treat all user-item interactions in the dataset as positive samples because having a rating implies that the user clicked on the item. Further, to ensure the training process goes smoothly with both positive and negative samples, we sample 4 negative samples for each positive sample in the training set and 99 negative samples for each positive sample in the test set.

4.1.2 Baselines. The baselines we select are as follows:

- · Static Recommendation Models. DIN [60], GRU4Rec [12], SASRec [17], and BERT4Rec [34] are all highly prevalent sequential recommendation methods in both academic research and the industry. They each incorporate different techniques, such as Attention, GRU (Gated Recurrent Unit), and SelfAttention, to enhance the recommendation process.
- · Dynamic Recommendation Models. DUET [29] and APG [48] consists of two parts: a parameter generation model and a primary model. The primary model refers to the aforementioned models like DIN, GRU4Rec, SASRec, BERT4Rec, etc. After pre-training, the parameter generation model can generate model parameters for the primary model during inference based on the samples.

4.1.3 Evaluation Metrics. We use the widely adopted AUC UAUC , , NDCG , and Recall as the metrics to evaluate model performance.

Algorithm 1: Pseudo code of SOLID

## Module 1: ▷ Item to Semantic

↦

Target : Item Sequence 𝑠 𝑣 → Semantic Sequence 𝑠 𝑐 Input : Item Sequence 𝑠 𝑣 Output : Semantic Sequence 𝑠 𝑐

## Module 2: ▷ Semantic Parameter Generation

↦

Target : Semantic Sequence 𝑠 𝑐 → Semantic Parameter Generator M 𝑐 and Semantic Encoder 𝐸 𝑐 Input : Semantic Sequence 𝑠 𝑐 Output : (Parameter Θ 𝑑 ), Prediction ˆ 𝑦

## Module 3: ▷ Semantic Metacode Learning

↦

Target : Item Sequence 𝑠 𝑣 , Semantic Sequence 𝑠 𝑐 → Item Pa- rameter Generator M 𝑣 , Item Encoder 𝐸 𝑣 , Semantic Parameter Generator M 𝑐 , and Semantic Encoder 𝐸 𝑐 Input : Item Sequence 𝑠 𝑣 , Semantic Sequence 𝑠 𝑐 Output : (Parameter Θ 𝑑 ), Prediction ˆ 𝑦

## Module 4: ▷ Semantic Codebook Learning

↦

Target : Item Sequence 𝑠 𝑣 , Semantic Sequence 𝑠 𝑐 , Semantic Encoder 𝐸 𝑐 → Codebook 𝐷 Input : Item Sequence 𝑠 𝑣 , Semantic Sequence 𝑠 𝑐 , (Semantic Encoder 𝐸 𝑐 ) Output : (Parameter Θ 𝑑 ), Prediction ˆ 𝑦

Overview:

▷ Training Procedure

Input : Item Sequence 𝑠 𝑣 , Semantic Sequence 𝑠 𝑐 . Output : (Parameters Θ 𝑑 ), Prediction ˆ. 𝑦 Initialization : Randomly initialize the models M M , 𝑐 , M 𝑣 with parameters Θ 𝑠 , Θ 𝑐 , Θ 𝑣 respectively.

↦

Item Sequence 𝑠 𝑣 → Semantic Sequence 𝑠 𝑐

## repeat

if M 𝑐 and 𝐸 𝑐 have not yet been well-trained then Train as Eq.12

end until Convergence ;

Initialization : Initialize 𝐷 via pretrained 𝐸 𝑐

repeat if M 𝑐 and 𝐸 𝑐 have not yet been well-trained then Train as Eq.13

end until Convergence ;

return M M 𝑐 , 𝑣 , 𝐷 .

## 4.2 Overall Results

As shown in Table 1, we evaluate the overall performance across four multimodal datasets: Arts, Instruments, Office, and Scientific. For each dataset, we test the performance of four SR Models: DIN, GRU4Rec, SASRec, and BERT4Rec. We evaluate performance via AUC, UAUC, NDCG@10, Recall@10, NDCG@20, and Recall@20. For each SR Model, there are five options for DSR Models: None ('-'), APG, Ours (APG), DUET, and Ours (DUET), where '-' indicates no DSR Model usage, i.e., the inherent performance of the SR Model itself. Since the '-' option consistently performs worse than using a DSR Model, our comparison primarily focuses on the performance of APG vs. Ours (APG) and DUET vs. Ours (DUET)

Table 1: Performance comparison of the proposed method and baselines. The best results is in bold.

| Arts     | Arts                              | Arts                               | Arts                               | Arts                               | Arts                               | Arts                               | Arts                                      | Instruments   | Instruments                             | Instruments                        | Instruments                        | Instruments                        | Instruments                        | Instruments                        | Instruments                        |
|----------|-----------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|-------------------------------------------|---------------|-----------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|
| SR Model | DSR Model                         |                                    |                                    |                                    |                                    |                                    |                                           | SR Model      | DSR Model                               |                                    |                                    |                                    |                                    |                                    |                                    |
| SR Model | DSR Model                         | AUC                                | UAUC                               | NDCG@10                            | Recall@10                          | NDCG@20                            | Recall@20                                 | SR Model      | DSR Model                               | AUC                                | UAUC                               | NDCG@10                            | Recall@10                          | NDCG@20                            | Recall@20                          |
| DIN      | - APG Ours (APG) DUET Ours (DUET) | 0.8193 0.8432 0.8459 0.8338 0.8426 | 0.7559 0.7786 0.7873 0.7647 0.7830 | 0.2646 0.2868 0.2907 0.2837 0.3014 | 0.4696 0.5024 0.5144 0.4893 0.5162 | 0.2993 0.3221 0.3271 0.3185 0.3363 | 0.6054 0.6363 0.6529 0.6202 0.6486 0.6380 | DIN           | - APG Ours (APG) DUET Ours (DUET)       | 0.7974 0.8183 0.8274 0.8126 0.8207 | 0.7463 0.7534 0.7769 0.7499 0.7613 | 0.2620 0.2680 0.2918 0.2727 0.2850 | 0.4576 0.4606 0.5006 0.4658 0.4885 | 0.2966 0.3025 0.3257 0.3060 0.3183 | 0.5991 0.5962 0.6364 0.5970 0.6181 |
| GRU4Rec  | - APG Ours (APG) DUET Ours (DUET) | 0.8434 0.8416 0.8463 0.8463 0.8466 | 0.7837 0.7796 0.7897 0.7809 0.7915 | 0.2799 0.2828 0.3023 0.2911 0.3111 | 0.4943 0.4986 0.5242 0.5061 0.5368 | 0.3169 0.3196 0.3378 0.3277 0.3460 | 0.6403 0.6589 0.6430 0.6694               | GRU4Rec       | - APG Ours (APG) DUET Ours (DUET)       | 0.8103 0.8171 0.8296 0.8236 0.8261 | 0.7604 0.7578 0.7752 0.7568 0.7740 | 0.2770 0.2746 0.2911 0.2699 0.2958 | 0.4772 0.4716 0.4971 0.4655 0.4987 | 0.3102 0.3089 0.3265 0.3058 0.3313 | 0.6105 0.6069 0.6360 0.6059 0.6401 |
| SASRec   | - APG Ours (APG) DUET Ours (DUET) | 0.8383 0.8370 0.8414 0.8345 0.8469 | 0.7737 0.7687 0.7820 0.7660 0.7867 | 0.2758 0.2816 0.3018 0.2727 0.3022 | 0.4852 0.4884 0.5145 0.4763 0.5216 | 0.3127 0.3166 0.3365 0.3101 0.3382 | 0.6273 0.6222 0.6468 0.6177 0.6560        | SASRec        | - APG Ours (APG) DUET Ours (DUET) - APG | 0.8201 0.8200 0.8234 0.8241 0.8270 | 0.7586 0.7523 0.7573 0.7599 0.7661 | 0.2729 0.2663 0.2699 0.2768 0.2843 | 0.4705 0.4601 0.4622 0.4760 0.4827 | 0.3071 0.3010 0.3065 0.3105 0.3198 | 0.6051 0.5929 0.6029 0.6076 0.6206 |

| Office   | Office                            | Office                             | Office                             | Office                             | Office                             | Office                             | Office                             | Scientific   | Scientific                        | Scientific                         | Scientific                         | Scientific                                | Scientific                         | Scientific                         | Scientific                         |
|----------|-----------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|--------------|-----------------------------------|------------------------------------|------------------------------------|-------------------------------------------|------------------------------------|------------------------------------|------------------------------------|
| SR Model | DSR Model                         |                                    |                                    |                                    |                                    |                                    |                                    | SR Model     | DSR Model                         |                                    |                                    |                                           |                                    |                                    |                                    |
| SR Model | DSR Model                         | AUC                                | UAUC                               | NDCG@10                            | Recall@10                          | NDCG@20                            | Recall@20                          | SR Model     | DSR Model                         | AUC                                | UAUC                               | NDCG@10                                   | Recall@10                          | NDCG@20                            | Recall@20                          |
| DIN      | - APG Ours (APG) DUET Ours (DUET) | 0.8158 0.8359 0.8394 0.8297 0.8361 | 0.7510 0.7639 0.7673 0.7531 0.7642 | 0.2701 0.2862 0.2764 0.2813 0.2949 | 0.4702 0.4903 0.4823 0.4816 0.4970 | 0.3046 0.3202 0.3128 0.3147 0.3282 | 0.6045 0.6202 0.6222 0.6085 0.6240 | DIN          | - APG Ours (APG) DUET Ours (DUET) | 0.6100 0.7310 0.7315 0.6714 0.7138 | 0.5971 0.6969 0.6989 0.6266 0.6682 | 0.1337 0.1700 0.1746 0.1428 0.1589        | 0.2609 0.3238 0.3429 0.2736 0.3012 | 0.1648 0.2099 0.2147 0.1748 0.1989 | 0.3880 0.4816 0.5020 0.3979 0.4573 |
| GRU4Rec  | - APG Ours (APG) DUET Ours (DUET) | 0.8346 0.8343 0.8354 0.8399 0.8437 | 0.7606 0.7623 0.7671 0.7649 0.7737 | 0.2704 0.2809 0.2914 0.2930 0.3072 | 0.4762 0.4831 0.4966 0.4976 0.5112 | 0.3055 0.3154 0.3255 0.3268 0.3403 | 0.6117 0.6159 0.6272 0.6262 0.6366 | GRU4Rec      | - APG Ours (APG) DUET Ours (DUET) | 0.7424 0.7273 0.7402 0.7270 0.7410 | 0.7094 0.6933 0.7133 0.6881 0.7054 | 0.1621 0.1592 0.1859 0.1658 0.1792        | 0.3214 0.3159 0.3535 0.3224 0.3415 | 0.2049 0.1988 0.2273 0.2036 0.2196 | 0.4952 0.4758 0.5161 0.4703 0.5020 |
| SASRec   | - APG Ours (APG) DUET Ours (DUET) | 0.8288 0.8377 0.8402 0.8395 0.8460 | 0.7587 0.7603 0.7679 0.7594 0.7735 | 0.2820 0.2823 0.2997 0.2833 0.2997 | 0.4858 0.4804 0.4995 0.4831 0.5061 | 0.3153 0.3170 0.3333 0.3173 0.3345 | 0.6151 0.6117 0.6269 0.6105 0.6380 | SASRec       | - APG Ours (APG) DUET Ours (DUET) | 0.7175 0.6952 0.7161 0.6992 0.7111 | 0.6772 0.6610 0.6728 0.6565 0.6738 | 0.1587 0.1523 0.1634 0.1579 0.1548 0.1744 | 0.3145 0.3040 0.3122 0.3040 0.3016 | 0.1960 0.1910 0.2002 0.1944 0.1957 | 0.4631 0.4583 0.4580 0.4481 0.4614 |

for each SR Model. Across all datasets, all SR Models, and all metrics, our proposed methods significantly outperform both APG and DUET. We conducted experiments on four other commonly used recommendation datasets and compared the UAUC metric in Figures 3 and 4. Our method ({SR=SASRec, DSR=DUET}) significantly outperforms other SR and DSR Models across all the datasets.

Figure 3: UAUC comparison of the proposed method and baseline on the CDs and Electronic datasets.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00123v1-figure-71922893-d31e-4503-9e7b-9ea8ce708f55.jpg)

Figure 4: UAUC comparison of the proposed method and baseline on the Book and Music datasets.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00123v1-figure-3dd0a6c2-931a-4ffc-a31a-f30bd5bf41c9.jpg)

## 4.3 Ablation Study

We conduct ablation studies on each dataset, each SR and each DSR to further analyze the impact of modules and modalities. The ablation results on each dataset, DR, and DSR combinations are similar, so we only show the results under the condition {Dataset=Arts, SR=SASRec, DSR=DUET}. Each row's ! and % respectively indicate with and without the module/modality.

4.3.1 Ablation Study on Modules. As shown in Table 2, we conduct an ablation study on each module proposed in our method, SPG stands for Semantic Parameter Generation, SML stands for Semantic Metacode Learning, and SCL stands for Semancic Codebook Learning. Since SPG is a prerequisite for SML, SML cannot exist independently of SPG; therefore, there is no separate performance data for SML alone in the table. The first line represents the traditional DSR model where parameters are generated using an item sequence. The second line represents generating parameters using a semantic sequence. The third line represents the joint generation of parameters using both item sequence and semantic sequence, with joint training. The fourth line represents using semantic codebook learning without using semantic information. The fifth line represents our complete method. The experiments show that the model performs best when all three modules are used. In terms of individual modules, SCL has the greatest impact on performance.

Table 2: Results of the ablation study over our proposed methods with respect to the modules. The best results is in bold.

| Module    | Module    | Module    | Metrics                            | Metrics                            | Metrics                            | Metrics                            | Metrics                            | Metrics                            |
|-----------|-----------|-----------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|
| SPG       | SML       | SCL       | AUC                                | UAUC                               | NDCG@10                            | Recall@10                          | NDCG@20                            | Recall@20                          |
| % ! ! % ! | % % ! % ! | % % % ! ! | 0.8345 0.8459 0.8270 0.8461 0.8469 | 0.7660 0.7783 0.7530 0.7828 0.7867 | 0.2727 0.2905 0.2491 0.2979 0.3022 | 0.4763 0.5069 0.4539 0.5166 0.5216 | 0.3101 0.3270 0.2857 0.3326 0.3382 | 0.6177 0.6425 0.5922 0.6481 0.6560 |

4.3.2 Ablation Study on Modalities. As shown in Table 3, we conduct ablation study on each modality. The experimental results show that the fusion of three modalities-ID, Image, and Text-is not necessarily the best option. In terms of the impact on performance for individual modalities, Text &gt; Image &gt; ID. For the fusion of two modalities, in terms of impact on performance, ID + Text &gt; Image + Text &gt; ID + Image.

Table 3: Results of the ablation study over our proposed methods with respect to the modalities. The best results is in bold.

| Modality      | Modality      | Modality      | Metrics                                          | Metrics                                          | Metrics                                          | Metrics                                          | Metrics                                          | Metrics                                          |
|---------------|---------------|---------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|
| ID            | Image         | Text          | AUC                                              | UAUC                                             | NDCG@10                                          | Recall@10                                        | NDCG@20                                          | Recall@20                                        |
| ! % % ! ! % ! | % ! % ! % ! ! | % % ! % ! ! ! | 0.8479 0.8438 0.8480 0.8459 0.8490 0.8471 0.8469 | 0.7850 0.7818 0.7858 0.7832 0.7881 0.7857 0.7867 | 0.2983 0.2953 0.3031 0.2953 0.3016 0.2963 0.3022 | 0.5155 0.5117 0.5252 0.5148 0.5223 0.5173 0.5216 | 0.3347 0.3310 0.3379 0.3313 0.3376 0.3319 0.3382 | 0.6510 0.6476 0.6548 0.6492 0.6566 0.6513 0.6560 |

## 4.4 Depth Analysis

We further conduct depth analysis to demonstrate the effectiveness. Unless otherwise specified, the dataset, SR, and DSR default to Arts, SASRec, and DUET, respectively. Note that we get similar results for all settings, but only a subset of them are shown here.

4.4.1 Stability and Robustness. Wetested the variance of the UAUC for SOLID and DUET on each user in the Arts dataset when faced with similar user behaviors. Specifically, we added one user behavior at a time for each user behavior and calculated the performance variance. We then aggregated the variances for all users to obtain the median, mean, minimum, and maximum of these variances. Table 4 shows that SOLID has stronger stability and robustness compared to DUET.

Table 4: Variance comparison.

| DUET   | DUET   | DUET   | DUET   | Ours   | Ours   | Ours   | Ours   |
|--------|--------|--------|--------|--------|--------|--------|--------|
| Medium | Mean   | Min    | Max    | Medium | Mean   | Min    | Max    |
| 0.35   | 0.42   | 0.08   | 0.69   | 0.26   | 0.29   | 0.03   | 0.47   |

4.4.2 Cost Comparison. In Table 5, we do analysis based on the BERT4Rec (the biggest SR in our paper), the increased memory and time are not important because the increase is slight and does not affect real-time performance [29, 49].

Table 5: Cost of our method.

| DUET    | DUET            | DUET           | Ours    | Ours            | Ours           |
|---------|-----------------|----------------|---------|-----------------|----------------|
| #Param. | Train (s/epoch) | Test (s/batch) | #Param. | Train (s/epoch) | Test (s/batch) |
| 695.84k | 106.0106        | 0.0084         | 821.44k | 130.6742        | 0.0103         |

4.4.3 Hyperparameter Analysis. To analyze the impace of the main hyperparameters 𝜆 and T , we conduct grid search experiment. As shown in Figure 5, the horizontal axis represents 𝜆 , and the vertical axis represents T . The depth of the color and the radius of the circle represent the magnitude of the value; the larger the value, the deeper the color and the larger the circle (i.e., the larger the radius). Blue, green, and orange represent the metrics UAUC, NDCG@10, and Recall@10, respectively. The results show that the best performance is achieved when 𝜆 = 0 1 and . T = 0 01. .

Figure 5: Hyperparameter Grid Search.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00123v1-figure-e8296933-4871-4320-b50a-6c9158cf1061.jpg)

## 5 CONCLUSION

In this paper, we have presented the Semantic Codebook Learning for Dynamic Recommendation Models (SOLID) as a solution to the limitations faced by existing dynamic sequence recommendation systems (DSR). Our framework integrates multimodal information, including images and text, with user-item interactions to enhance recommendation accuracy and adaptability. By disentangling model parameters into trunk parameters capturing generalized user behavior trends and branch parameters tailored to individual user actions, SOLID offers a more efficient and effective recommendation system. Through extensive experimentation across multiple datasets, we have demonstrated that SOLID significantly outperforms previous DSR models, with an significant improvement on extensive datasets and models. These results underscore the potential of leveraging multimodal information to advance the capabilities of dynamic recommendation systems, paving the way for more personalized and responsive user experiences in the era of digital personalization.

## ACKNOWLEDGEMENTS

National Science and Technology Major Project (2022ZD0119100), Scientific Research Fund of Zhejiang Provincial Education Department (Y202353679), Key Research and Development Program of Zhejiang Province (No. 2024C03270), National Natural Science Foundation of China (No. 62441605), the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SNZJU-SIAS-0010).

## REFERENCES

- [1] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. 2022. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18511-18521.
- [2] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1798-1828.
- [3] Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. 2020. Controllable multi-interest framework for recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining . 2942-2951.
- [4] Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2021. Sequential recommendation with graph neural networks. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval . 378-387.
- [5] Yongjun Chen, Zhiwei Liu, Jia Li, Julian McAuley, and Caiming Xiong. 2022. Intent contrastive learning for sequential recommendation. In Proceedings of the ACM Web Conference 2022 . 2172-2182.
- [6] Zhengyu Chen, Teng Xiao, Kun Kuang, Zheqi Lv, Min Zhang, Jinluan Yang, Chengqiang Lu, Hongxia Yang, and Fei Wu. 2024. Learning to Reweight for Generalizable Graph Neural Network. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 8320-8328.
- [7] Zhengyu Chen, Ziqing Xu, and Donglin Wang. 2021. Deep transfer tensor decomposition with orthogonal constraint for recommender systems. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 4010-4018.
- [8] Tan M Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. 2022. Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11389-11398.
- [9] Junhao Feng, Guohua Wang, Changmeng Zheng, Yi Cai, Ze Fu, Yaowei Wang, Xiao-Yong Wei, and Qing Li. 2023. Towards bridged vision and language: Learning cross-modal knowledge representation for relation extraction. IEEE Transactions on Circuits and Systems for Video Technology 34, 1 (2023), 561-575.
- [10] Kairui Fu, Shengyu Zhang, Zheqi Lv, Jingyuan Chen, and Jiwei Li. 2024. DIET: Customized Slimming for Incompatible Networks in Sequential Recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining .
- [11] David Ha, Andrew M. Dai, and Quoc V. Le. 2017. HyperNetworks. In 5th International Conference on Learning Representations, ICLR 2017 .
- [12] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based recommendations with recurrent neural networks. International Conference on Learning Representations 2016 (2016).
- [13] Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster) 3 (2017).
- [14] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023. Make-an-audio: Textto-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning . PMLR, 13916-13932.
- [15] Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. 2022. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204.09934 (2022).
- [16] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. 2016. Dynamic filter networks. Advances in neural information processing systems 29 (2016).
- [17] Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM) . IEEE, 197-206.
- [18] Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
- [19] Haoxuan Li, Chunyuan Zheng, Yixiao Cao, Zhi Geng, Yue Liu, and Peng Wu. 2023. Trustworthy policy learning under the counterfactual no-harm criterion. In International Conference on Machine Learning . PMLR, 20575-20598.
- [20] Haoxuan Li, Chunyuan Zheng, Peng Wu, Kun Kuang, Yue Liu, and Peng Cui. 2023. Who should be given incentives? counterfactual optimal treatment regimes
- learning for recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 1235-1247.
- [21] Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. 2022. Fine-grained semantically aligned vision-language pre-training. Advances in neural information processing systems 35 (2022), 7290-7303.
- [22] Xinting Liao, Weiming Liu, Xiaolin Zheng, Binhui Yao, and Chaochao Chen. 2023. Ppgencdr: A stable and robust framework for privacy-preserving cross-domain recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 4453-4461.
- [23] Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and TatSeng Chua. 2024. Data-efficient Fine-tuning for LLM-based Recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 365-374.
- [24] Xinyu Lin, Wenjie Wang, Jujia Zhao, Yongqi Li, Fuli Feng, and Tat-Seng Chua. 2024. Temporally and distributionally robust optimization for cold-start recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 8750-8758.
- [25] Weiming Liu, Chaochao Chen, Xinting Liao, Mengling Hu, Yanchao Tan, Fan Wang, Xiaolin Zheng, and Yew Soon Ong. 2024. Learning Accurate and Bidirectional Transformation via Dynamic Embedding Transportation for Cross-Domain Recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 8815-8823.
- [26] Weiming Liu, Xiaolin Zheng, Chaochao Chen, Jiajie Su, Xinting Liao, Mengling Hu, and Yanchao Tan. 2023. Joint internal multi-interest exploration and external domain alignment for cross domain sequential recommendation. In Proceedings of the ACM Web Conference 2023 . 383-394.
- [27] Zheqi Lv, Feng Wang, Shengyu Zhang, Wenqiao Zhang, Kun Kuang, and Fei Wu. 2023. Parameters Efficient Fine-Tuning for Long-Tailed Sequential Recommendation. In CAAI International Conference on Artificial Intelligence . Springer, 442-459.
- [28] Zheqi Lv, Wenqiao Zhang, Zhengyu Chen, Shengyu Zhang, and Kun Kuang. 2024. Intelligent model update strategy for sequential recommendation. In Proceedings of the ACM on Web Conference 2024 . 3117-3128.
- [29] Zheqi Lv, Wenqiao Zhang, Shengyu Zhang, Kun Kuang, Feng Wang, Yongwei Wang, Zhengyu Chen, Tao Shen, Hongxia Yang, Beng Chin Ooi, and Fei Wu. 2023. DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalization. In Proceedings of the ACM Web Conference 2023 .
- [30] Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Xin Wang, and Wenwu Zhu. 2020. Disentangled self-supervision in sequential recommenders. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining . 483-491.
- [31] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. 2021. Personalized federated learning using hypernetworks. In International Conference on Machine Learning . PMLR, 9489-9502.
- [32] Jiajie Su, Chaochao Chen, Zibin Lin, Xi Li, Weiming Liu, and Xiaolin Zheng. 2023. Personalized behavior-aware transformer for multi-behavior sequential recommendation. In Proceedings of the 31st ACM International Conference on Multimedia . 6321-6331.
- [33] Jiajie Su, Chaochao Chen, Weiming Liu, Fei Wu, Xiaolin Zheng, and Haoming Lyu. 2023. Enhancing hierarchy-aware graph networks with deep dual clustering for session-based recommendation. In Proceedings of the ACM Web Conference 2023 . 165-176.
- [34] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management . 1441-1450.
- [35] Zihao Tang, Zheqi Lv, Shengyu Zhang, Fei Wu, and Kun Kuang. 2024. ModelGPT: Unleashing LLM's Capabilities for Tailored Model Generation. arXiv preprint arXiv:2402.12408 (2024).
- [36] Zihao Tang, Shengyu Zhang, Zheqi Lv, Yifan Zhou, Xinyu Duan, Kun Kuang, and Fei Wu. 2024. AuG-KD: Anchor-Based Mixup Generation for Out-ofDomain Knowledge Distillation. In 12th International Conference on Learning Representations, ICLR 2024, Vienna Austria, May 7-11, 2024 . OpenReview.net. https://openreview.net/forum?id=fcqWJ8JgMR
- [37] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. 2020. Continual learning with hypernetworks. In 8th International Conference on Learning Representations, ICLR 2020 .
- [38] Jiawei Wang, Yuquan Le, Da Cao, Shaofei Lu, Zhe Quan, and Meng Wang. 2024. Graph Reasoning With Supervised Contrastive Learning for Legal Judgment Prediction. IEEE Transactions on Neural Networks and Learning Systems (2024).
- [39] Jiawei Wang, Zhanchang Ma, Da Cao, Yuquan Le, Junbin Xiao, and Tat-Seng Chua. 2023. Deconfounded Multimodal Learning for Spatio-temporal Video Grounding. In Proceedings of the 31st ACM International Conference on Multimedia . 7521-7529.
- [40] Jinpeng Wang, Ziyun Zeng, Yunxiao Wang, Yuting Wang, Xingyu Lu, Tianxiang Li, Jun Yuan, Rui Zhang, Hai-Tao Zheng, and Shu-Tao Xia. 2023. Missrec: Pretraining and transferring multi-modal interest-aware sequence representation

for recommendation. In Proceedings of the 31st ACM International Conference on Multimedia . 6548-6557.

- [41] Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. 2022. Disentangled representation learning. arXiv preprint arXiv:2211.11695 (2022).
- [42] Xin Wang, Hong Chen, Yuwei Zhou, Jianxin Ma, and Wenwu Zhu. 2022. Disentangled representation learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 1 (2022), 408-424.

[43] Xin Wang, Hong Chen, and Wenwu Zhu. 2021. Multimodal disentangled representation for recommendation. In 2021 IEEE International Conference on Multimedia and Expo (ICME) . IEEE, 1-6.

[44] Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Ge, and Wenwu Zhu. 2023. Curriculum co-disentangled representation learning across multiple environments for social recommendation. In International Conference on Machine Learning . PMLR, 36174-36192.

[45] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019. Session-based recommendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , Vol. 33. 346-353.

[46] Zhou Xian, Shamit Lal, Hsiao-Yu Tung, Emmanouil Antonios Platanios, and Katerina Fragkiadaki. 2021. HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworks. In 9th International Conference on Learning Representations, ICLR 2021 .

[47] Teng Xiao, Zhengyu Chen, and Suhang Wang. 2023. Reconsidering Learning Objectives in Unbiased Recommendation: A Distribution Shift Perspective. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 2764-2775.

[48] Bencheng Yan, Pengjie Wang, Kai Zhang, Feng Li, Jian Xu, and Bo Zheng. 2022. APG: Adaptive Parameter Generation Network for Click-Through Rate Prediction. In Advances in Neural Information Processing Systems .

[49] Jiangchao Yao, Feng Wang, Xichen Ding, Shaohu Chen, Bo Han, Jingren Zhou, and Hongxia Yang. 2022. Device-cloud Collaborative Recommendation via Meta Controller. In KDD'22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022 . 4353-4362.

[50] Chris Zhang, Mengye Ren, and Raquel Urtasun. 2019. Graph HyperNetworks for Neural Architecture Search. In 7th International Conference on Learning Representations, ICLR 2019 .

[51] Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang. 2021. Mining latent structures for multimedia recommendation. In Proceedings of the 29th ACM international conference on multimedia . 3872-3880.

[52] Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Mengqi Zhang, Shu Wu, and Liang Wang. 2022. Latent structure mining with contrastive modality fusion for multimedia recommendation. IEEE Transactions on Knowledge and Data Engineering 35, 9 (2022), 9154-9167.

[53] Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, et al. 2024. HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models. arXiv preprint arXiv:2403.13447 (2024).

[54] Wenqiao Zhang and Zheqi Lv. 2024. Revisiting the domain shift and sample uncertainty in multi-source active domain transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 16751-16761.

[55] Wenqiao Zhang, Haochen Shi, Siliang Tang, Jun Xiao, Qiang Yu, and Yueting Zhuang. 2021. Consensus graph representation learning for better grounded image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 3394-3402.

[56] Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal intervention for leveraging popularity bias in recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval . 11-20.

[57] Yang Zhang, Tianhao Shi, Fuli Feng, Wenjie Wang, Dingxian Wang, Xiangnan He, and Yongdong Zhang. 2023. Reformulating CTR Prediction: Learning Invariant Feature Interactions for Recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1386-1395.

[58] Yin Zhang, Ziwei Zhu, Yun He, and James Caverlee. 2020. Content-collaborative disentanglement representation learning for enhanced recommendation. In Proceedings of the 14th ACM Conference on Recommender Systems . 43-52.

[59] Changmeng Zheng, Junhao Feng, Yi Cai, Xiaoyong Wei, and Qing Li. 2023. Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 6810-6824.

[60] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining . 1059-1068.

[61] Didi Zhu, Yinchuan Li, Min Zhang, Junkun Yuan, Jiashuo Liu, Kun Kuang, and Chao Wu. 2023. Bridging the gap: neural collapse inspired prompt tuning for generalization under class imbalance. arXiv preprint arXiv:2306.15955 (2023).

- [62] Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. 2024. Model Tailor: Mitigating Catastrophic Forgetting in Multimodal Large Language Models. arXiv preprint arXiv:2402.12048 (2024).
- [63] Yun Zhu, Haizhou Shi, Zhenshuo Zhang, and Siliang Tang. 2024. Mario: Model agnostic recipe for improving ood generalization of graph contrastive learning. In Proceedings of the ACM on Web Conference 2024 . 300-311.
- [64] Yun Zhu, Yaoke Wang, Haizhou Shi, and Siliang Tang. 2024. Efficient tuning and inference for large language models on textual graphs. arXiv preprint arXiv:2401.15569 (2024).
- [65] Yun Zhu, Yaoke Wang, Haizhou Shi, Zhenshuo Zhang, Dian Jiao, and Siliang Tang. 2024. GraphControl: Adding Conditional Control to Universal Graph Pretrained Models for Graph Domain Transfer Learning. In Proceedings of the ACM on Web Conference 2024 . 539-550.

## A APPENDIX

This is the Appendix for 'Semantic Codebook Learning for Dynamic Recommendation Models'.

## A.1 Supplementary Experiments

A.1.1 Datasets. The statistics of the datasets used in the experiments is shown in Table 6.

## Table 6: Statistics of Datasets.

| Dataset     | #User     | #Item   | #Interaction   |   Density |
|-------------|-----------|---------|----------------|-----------|
| Arts        | 45,486    | 21,019  | 395,150        | 0.0004133 |
| Office      | 87,436    | 25,986  | 684,837        | 0.0003014 |
| Instruments | 24,962    | 9,964   | 208,926        | 0.00084   |
| Scientific  | 8,442     | 4,385   | 59,427         | 0.0016053 |
| CDs         | 1,578,597 | 486,360 | 3,749,004      | 4.9e-06   |
| Electronic  | 4,201,696 | 476,002 | 7,824,482      | 3.9e-06   |
| Book        | 46,549    | 212,996 | 1,861,533      | 0.0001878 |
| Music       | 39,743    | 164,224 | 1,792,502      | 0.0002746 |

A.1.2 Hyperparameters and Training Schedules. Wesummarize the hyperparameters and training schedules of the datasets used in the experiments in Table 7.

Table 7: Hyperparameters and training schedules of SOLID.

| Dataset                                                      | Parameters                                                                                                                                                                 | Setting                                    |
|--------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------|
| Arts Office Instruments Scientific CDs Electronic Book Music | GPU Optimizer Learning Rate Batch Size Sequence Length the Dimension of Embedding the Amount of MLP Hidden Dimension of Semantic Codebook z Dimension of Semantic Codebook | Tesla A100 Adam 0.001 1024 10 1×32 2 64 32 |
Zheqi Lv Shaoxuan He Tianyu Zhan Shengyu Zhang Wenqiao Zhang Jingyuan Chen Zhou Zhao Fei Wu
Semantic Codebook Learning for Dynamic Recommendation Models
Dynamic sequential recommendation (DSR) can generate model parameters based
on user behavior to improve the personalization of sequential recommendation
under various user preferences. However, it faces the challenges of large
parameter search space and sparse and noisy user-item interactions, which
reduces the applicability of the generated model parameters. The Semantic
Codebook Learning for Dynamic Recommendation Models (SOLID) framework presents
a significant advancement in DSR by effectively tackling these challenges. By
transforming item sequences into semantic sequences and employing a dual
parameter model, SOLID compresses the parameter generation search space and
leverages homogeneity within the recommendation system. The introduction of the
semantic metacode and semantic codebook, which stores disentangled item
representations, ensures robust and accurate parameter generation. Extensive
experiments demonstrates that SOLID consistently outperforms existing DSR,
delivering more accurate, stable, and robust recommendations.