## Deep Uncertainty-Based Explore for Index Construction and Retrieval in Recommendation System

Xin Jiang Shopee Discovery Ads Beijing, China andy.jiang@shopee.com

Kaiqiang Wang Shopee Discovery Ads Beijing, China kaiqiang.wang@shopee.com

Yinlong Wang Shopee Discovery Ads Beijing, China yinlong.wang@shopee.com

Fengchang Lv Shopee Discovery Ads Beijing, China fengchang.lv@shopee.com

Xianteng Wu Shopee Discovery Ads Beijing, China xianteng.wu@shopee.com

## Taiyang Peng

Shopee Discovery Ads Beijing, China taiyang.peng@shopee.com

Pengye Zhang Shopee Discovery Ads Beijing, China pengye.zhang@shopee.com

## Yifan Zeng

Shopee Discovery Ads Beijing, China yifan.zeng@shopee.com

## ABSTRACT

## 1 INTRODUCTION

In recommendation systems, the relevance and novelty of the final results are selected through a cascade system of Matching -&gt; Ranking -&gt; Strategy. The matching model serves as the starting point of the pipeline and determines the upper bound of the subsequent stages. Balancing the relevance and novelty of matching results is a crucial step in the design and optimization of recommendation systems, contributing significantly to improving recommendation quality. However, the typical matching algorithms have not simultaneously addressed the relevance and novelty perfectly. One main reason is that deep matching algorithms exhibit significant uncertainty when estimating items in the long tail (e.g., due to insufficient training samples) items.

The uncertainty not only affects the training of the models but also influences the confidence in the index construction and beam search retrieval process of these models.

This paper proposes the UICR ( U ncertainty-based explore for I ndex C onstruction and R etrieval) algorithm, which introduces the concept of uncertainty modeling in the matching stage and achieves multi-task modeling of model uncertainty and index uncertainty. The final matching results are obtained by combining the relevance score and uncertainty score infered by the model. Experimental results demonstrate that the UICR improves novelty without sacrificing relevance on real-world industrial productive environments and multiple open-source datasets. Remarkably, online A/B test results of display advertising in Shopee demonstrates the effectiveness of the proposed algorithm.

## KEYWORDS

Uncertainty, Model-based Retrieval, Matching, Recommendation System

Recommendation systems adopt a multi-stage cascade architecture [5, 11], primarily comprising matching, ranking and strategy. During the matching stage[23], thousands of candidates are retrieved from an extensive corpus. In the subsequent ranking and strategy stages, these retrieved candidates are prioritized based on user preferences and business rules. One of the core objectives of recommendation systems is to deliver recommendations that are not only relevant to users but also exhibit novelty [6, 14, 17, 26]. Novelty typically refers to items or content recommended to users that they have not encountered in their historical behavior or exposure. It is important to note that the set of items considered novel varies for different users. Therefore, at the forefront of the system pipeline, in the matching stage, it is crucial to balance the relevance and the novelty.

In industrial-scale recommendation systems, the matching service based on deep models, such as [2, 3, 10], often needs to retrieve thousands of items from millions to billions of candidate items within tens of milliseconds for subsequent scoring in the downstream pipeline. To accelerate the matching process, recommendation systems often employ a data structure called an index, such as the HNSW[15] algorithm, to assist the matching model in storing and retrieving candidate items relevant to user interests.

During the training process of deep matching models, due to the lack of sufficient training samples for long-tail items [7, 13, 25], traditional point estimation training methods fail to adequately train the embeddings for these items. This leads to two problems: 1) Under traditional point estimation modeling, these models exhibit significant uncertainty in predicting long-tail items (e.g., due to insufficient sample size[16, 18, 22]). 2) The index, constructed by

Shuai Yang Shopee Discovery Ads Beijing, China lucas.yang@shopee.com

## Shuo Yuan

Shopee Discovery Ads Beijing, China shuoyuan.ys@shopee.com

calculation upon insufficient trained embeddings, is lack uncertainty information, thus unable to provide high relevance and low uncertainty items set to the deep matching model, resulting in a loss of relevance in the final matching results. Uncertainty refers to the degree of uncertainty in the recommendation system's predictions or estimates of user interests, preferences, or behavior. It reflects the system's understanding of users and the reliability of the recommendation results. Therefore we introduce variance and transform the problem from a point estimation of the user interest to a distribution estimate, use variance to characterize the magnitude of uncertainty. At the same time, these matching algorithms focus more on improving relevance and pay insufficient attention to novelty metrics. In some cases, the improvement in relevance metrics comes at the expense of novelty metrics, which is not conducive to the long-term ecological development of recommendation systems.

To address the aforementioned issues, this paper proposes the UICRmethod. UICR comprises three main components: Uncertaintybased Index (UN-Index), Uncertainty-based Retrieval (UN-Retrieval), and Uncertainty-based Model Training for Index and Retrieval (UNModel). UN-Index is responsible for constructing a high-confidence index, providing a high-quality index structure for matching. UNRetrieval, building upon UN-Index, integrates relevance scores and uncertainty scores during the retrieval process to balance the relevance and novelty of matching results. UN-Model provides modeling capabilities for estimating relevance scores and uncertainty scores for both UN-Index and UN-Retrieval.

The main contributions of this paper are as follows:

- · To improve the balance of relevance and novelty, this paper introduces a matching framework incorporating uncertainty, consisting of uncertainty-based index construction, uncertainty-based retrieval, and uncertainty-based modeling.
- · To the best of our knowledge, this paper proposes introducing uncertainty information during the index construction phase of the matching process to build a high-quality index for the first time.
- · On the deep matching direction based on indexing, UICR demonstrates a commendable performance in balancing relevance and novelty in recommendation systems.

## 2 METHODOLOGY

In this section, we first provide the problem definition in Sec. 2.1. Then, we introduce the pipeline for Uncertainty-based Index Construction and Retrieval(UICR) in Sec. 2.2. Next, we present the uncertainty-based index(UN-Index) construction method in Sec. 2.3. Then, we elaborate on the uncertainty-based retrieval(UN-Retrieval) algorithm in Sec. 2.4. Afterwards, we introduce the uncertainty estimate module, which will be utilized for user-to-item and itemto-item uncertainty estimation, in Sec. 2.5. Finally, we describe the uncertainty-based model training method in Sec. 2.6.

## 2.1 Problem Definition

The objective of matching phase in the Recommendation System is to retrieval a subset of items from the total candidate items, which may reach billion-scale. In this paper, we utilize I and U denote the item candidate set and user set respectively, 𝑖 and 𝑢 stand for a specific item 𝑖 ∈ I and user 𝑢 ∈ U . The matching model can be defined as M( 𝑥 𝑢 , 𝑥 𝑖 ) . It is a similarity mapping function according to some performance metrics.

The index in matching stage is defined as HI , which incorporates features of items in I and target to increace the retrieval efficiency of retrieval.

$$\mathcal { B } ( u ) = a r g T o p K _ { i \in I } \mathcal { F } ( \mathcal { M } ( x _ { u }, x _ { i } ), \mathcal { H } _ { I } ) \quad \quad ( 1 )$$

where F is the retrieval method based on matching model M and index H B( ) , 𝑢 is the retrieval outcomes for user 𝑢 𝑥 , 𝑢 and 𝑥 𝑖 are user and item features respectively.

## 2.2 Uncertainty-based Index Construction and Retrieval Pipeline

In this paragraph, we describe the pipeline for constructing a uncertainty-based matching model. The pipeline consists of three parts: UN-Index (an item-to-item uncertainty-based index construction algorithm), UN-Retrieval (an user-to-item uncertaintybased beam search retrieval algorithm) and UN-Model (a multi-task model training method that considers uncertainty of both user-toitem and item-to-item).

In order to simultaneously enhance relevance and novelty during the matching phase, the design concept of UICR involves introducing uncertainty to improve the matching results with high estimated scores and low uncertainty, thus enhancing the relevance of matching results. By increasing percentage the matching results with high uncertainty, it improves the novelty of the results. Specifically, UN-Index focuses on improving relevance. During the construction of UN-Index, only index paths with high estimated scores and low uncertainty are retained, while paths with high uncertainty are removed. UN-Retrieval emphasizes enhancing novelty. During the retrieval process, a measurement scheme based on uncertainty is used to select top 𝑘 for users. UN-Model provides corresponding uncertainty and score estimation capabilities for both aspects.

The workflow of the UICR is as follow:

- · Step 1 The UN-Model is trained to develop the ability to estimate user-item relevance, item-item similarities, and uncertainty estimation. This enables us to predict the relevance of &lt;user, item&gt; pairs and estimate the similarity between items.
- · Step 2 In the UN-Index module, the confidence index is constructed. Based on the item-item similarity and confidence, a high-confidence index is built.
- · Step 3 In the UN-Retrieval module, using the index structure generated by the UN-Index module and the &lt;user, item&gt; model provided by the UN-Model module, a multi-layered search is performed to select the top-K recommendations.

## 2.3 Uncertainty-based Index Construction Algorithm

2.3.1 Why do we need to model the uncertainty in index construction? A significant practical issue in the industry is that matching model often faces candidate item sets ranging from millions to billions. Therefore, a common solution in the industry is to build index structures to boost the model retrieval efficiency. As we know, HNSW index is one of the most widely used method in productive

Figure 1: Uncertainty-based Matching Modeling Pipeline

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00799v2-figure-431138e1-9593-46b0-8647-21448615f09d.jpg)

environments. During the construction of the HNSW index, only the distances between item-to-item embeddings are considered to select the neighboring items for each item.

However, in practical industrial systems, due to various reasons, the estimation of distances is often not very accurate. The construction of the index relies on the distances between &lt;item, item&gt;, which can lead to bias in the resulting index. Consequently, the constructed index may not be the optimal one, thereby reducing the accuracy of the beam search results during the matching stage. There are many reasons for inaccurate distance estimation, such as the imbalance in item exposure caused by the Matthew effect. This imbalance leads to insufficient embedding learning for items other than those in the head, resulting in certain uncertainties in distance calculation.

Therefore, in the index construction phase, we introduce uncertainty to enhance each item's ability to select its truly close neighboring items, aiming to improve the accuracy of index construction. Considering the uncertainty in the index and comparing with the definition in equation (1), matching model can be re-formulated as: B( 𝑢 ) = 𝑎𝑟𝑔𝑇𝑜𝑝𝐾 𝑖 ∈I F(M( 𝑥 𝑢 , 𝑥 𝑖 ) , H ′ ) (2)

$$1$$

where H ′ I is the index constructed by UN-Index.

2.3.2 Uncertainty-based Index Construction. In this section, we introduce the uncertainty-based HNSW index construction method. The standard HNSW index construction process does not take the uncertainty of distance of each item-to-item pair into the consideration, thus, the candidates of subsequent beam search retrieval can be improved by incorporate both the uncertainty-based modeling and the uncertainty-based distance weighted calculation. The fundamental reason for choosing the uncertainty-based distance weighted calculation in index construction is to decrease the uncertainty of the distance between the points in the candidate set and the seed points during the search process of the beam search algorithm. Meanwhile, due to the decrease of uncertainty in the distance measurement of item-to-item pairs among the index, this contributes to enhancing the relevance of retrieval results.

The calculation method of weighted distance between two item embeddings can be described as follow:

$$\stackrel { \cdot } { d i s t a n c e } ^ { ^ { \prime } } _ { j, k } = d i s t a n c e _ { j, k } \cdot \min ( m, ( 1 + \alpha \cdot v a r _ { j, k } ) ) \quad ( 3 ) \quad \text{eval}$$

where 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 ′ 𝑗,𝑘 is the weighted distance between item 𝑗 and item 𝑘 , 𝐿 2 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑗,𝑘 is the raw distance between item 𝑗 and item 𝑘 in HNSW index, 𝑣𝑎𝑟 𝑗,𝑘 is the variance of similarity between the embedding of item 𝑗 and item 𝑘 𝑚 , and 𝛼 are hyper parameters need to be tuned. Hyper parameter 𝛼 is a crucial parameter during the process of optimizing the constructed HNSW index and it should be set to larger than 0. According to equation (3), items with larger variances will have larger weighted distance values after weighting, while items with smaller variances will have smaller weighted distance values. The criterion for selecting which item will be retain is that the smaller the weighted distance, the nearer the item. In other words, items with closer weighted distances will be retained.

So the uncertainty-based index construction is implemented as follows,

- · Step 1 The standard HNSW index construction algorithm is used to build the original HNSW index. During the index construction process, the number of neighbours need to be saved of each point is set to 𝑛 .
- · Step 2 The item-to-item confidence module in the trained model is invoked to score each item-to-item pair and save the variance of each pair that are loaded from the HNSW index constructed in Step 1.
- · Step 3 We use the variance to weight distance between each item-to-item pair by equation (3) and the nearest 𝑛 ′ items, where 𝑛 ′ &lt; 𝑛 , in the HNSW index built in Step 1 are retained.

Thus, we will ultimately utilize the uncertainty-based index with a neighbor count of 𝑛 ′ for retrieval.

## 2.4 Uncertainty-based Retrieval Algorithm

2.4.1 Whydoweneedtoconsider the uncertainty in retrieval process. As shown in the Figure 2, items in the candidate set can be divided into four quadrants(The higher the variance (var), the greater the uncertainty). Items with high relevance and low uncertainty(low variance) are typically preferred for display, while items with low relevance and low uncertainty may not be prioritized for display. The remaining two quadrants, which consist of items with low relevance and high uncertainty, as well as items with high relevance

and high uncertainty, may benefit from appropriate exposure opportunities to enhance the system's estimation capability for their relevance and novelty. Therefore, in the retrieval process, we need to take into account both relevance and uncertainty to determine whether to return items from the candidate set to the downstream pipeline of the recommendation system.

We discuss three scenarios: 1) In a matching model that does not consider variances, the items selected by the matching model are located in the first and fourth quadrants, representing highly correlated items. 2) When we retain items with low variances of distance among neighbors and do not consider the user-to-item variance of the matching model, compared to the scenarios 1), the number of items in the fourth quadrant increases, while the number of items in the other three quadrants decreases. Therefore, from the perspective of relevance, the matching performance of the model improves. 3) In UICR, when the index retains item-to-item neighbors with low variances and the model retains items with high user-to-item variances, the number of items in the fourth quadrant can be slightly increased by adjusting the parameters that control the impact of variance on the weight scores. At the same time, the number of items retained in the first and second quadrants is increased, allowing the model to improve performance in terms of both correlation and novelty dimensions.

## Algorithm 1 UN Layer Search

Input: 𝑒𝑝 is the entry points

𝑙 𝑐

is the current layer number

𝑇 𝑐 is the number of steps to search in the current layer

1: 𝑆 ← 𝑒𝑝 //set of visited points

2: 𝐶 ← 𝑒𝑝 //set for candidates

3: 𝑊 ← 𝑒𝑝 //dynamic list of results

4: for 𝑡 = 1 → 𝑇 𝑐 do

5: 𝑁 ← union of neighbors at layer 𝑙 𝑐 of all items in 𝐶

6: 𝑁 ← 𝑁 - 𝑆

//pruning visited items

7: 𝑆 ← 𝑆 ∪ 𝑁 //mark as visited

8: 𝑊 ← 𝑎𝑟𝑔𝑇𝑜𝑝𝐾 𝑣 ∈ 𝑊 𝑈 ∪ 𝑆 𝑓 𝑢𝑠𝑖𝑜𝑛 ( 𝑢, e 𝑣 )

9: 𝐶 ← 𝑊 ∩ 𝑁 //new candidates

10: end for

11: return 𝑊

Output: 𝑒𝑓 𝑐 results with largest score s(u, v) to user 𝑢

2.4.2 Uncertainty-based beam search Algorithm. We enhance the original HNSW search algorithm by incorporating beam search with variance weighted calculation and propose a method called uncertainty-based beam search retrieval.

The search process of HNSW[15] entails navigating through a hierarchy of proximity graphs in a layer-wise and top-down fashion. Due to the utilization of the uncertainty-based distance weighted calculation approach during index construction, higher confidence and closer distance points are selected as neighbors, resulting in an improvement in the similarity of adjacent items during the index construction process. Consequently, the relevance of the items returned by the matching model is enhanced compared with considering only the relevance score of the user-to-item submodel. Thus, by incorporating uncertainty-based score weighted approach, we can merge the model's confidence scores for user-item pairs with the relevance scores as follow:

$$S _ { f u s i o n } = s c o r e _ { u 2 i } + \beta \cdot v a r _ { u 2 i } \quad \quad ( 4 ) \quad \text{ta}$$

where 𝛽 is a hyper-parameter needs to be tuned.

## 2.5 Uncertainty Estimate Module

An ideal uncertainty modeling module in UICR should have the capability of estimating the variance with high accuracy for each

Figure 2: Generally, for a user, all items in the candidate set can be divided into four quadrants based on their objectively real but numerically unknown relevances and variances.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00799v2-figure-b418aee8-dfcd-4e0f-87d4-85785e8c5ec6.jpg)

relevance score with minimal online time and space complexity, while ensuring no loss in relevance.

In terms of implementation details, the input of this module is the logit function, noted as 𝑓 , which can be formulated as follow:

$$f ( x _ { s }, x _ { t } ) \overset { \cdot } { = } \sigma ^ { - 1 } ( \mathcal { M } _ { s 2 t } ( x _ { s }, x _ { t } ) )$$

where M 𝑠 2 𝑡 is the sub-model of source-to-target, 𝑥 𝑠 and 𝑥 𝑡 are the source and target features. The module's output consists of two values: the estimated score and the variance. The uncertainty estimation module of source-to-target part can be represented by the following formula:

$$< s c o r e _ { s 2 t }, v a r _ { s 2 t } > = U E _ { s 2 t } ( f ( x _ { s }, x _ { t } ) ) \quad \quad ( 6 )$$

where 𝑈𝐸 𝑠 2 𝑡 is the Uncertainty Estimate module for source-totarget pair, 𝑠𝑐𝑜𝑟𝑒 𝑠 2 𝑡 and 𝑣𝑎𝑟 𝑠 2 𝑡 are the estimated relevance score and the corresponding variance. The loss function of the 𝑈𝐸 𝑠 2 𝑡 module is noted as L 𝑠 2 𝑡 . In the specific tasks afterwards, this module will be utilized to estimate user-to-item and item-to-item pairs.We will introduce the training method of the 𝑈𝐸 𝑠 2 𝑡 model in Sec. 2.6.

## 2.6 Model Training

2.6.1 Uncertainty modeling algorithm. According to the uncertaintybased index construction and beam search methods introduced in Sec. 2.3 and Sec. 2.4, it is obviously that the uncertainty modeling module required by the UICR algorithm needs to satisfy the following requirements: 1) The uncertainty modeling module needs to have low time and space complexity ,because the uncertainty-based beam search for user-to-item needs to be deployed as online services for the recommendation system. 2) The uncertainty modeling module needs to have strong model architectural compatibility, because in real productive environment, the matching model includes complex model structures, such as long sequence model structures, multi-objective model structures, multi-interest model structures, etc. 3) The uncertainty modeling module should be independent, capable of separately modeling the uncertainty of user-to-item and item-to-item, and each module should be able to work independently in inference calculations and uncertainty estimation.

In the UICR framework, Deep Uncertainty-Aware Learning [4] is the method we choose to model the user-to-item and item-to-item uncertainty. The UICR framework does not require any specific method here for the uncertainty modeling. Therefore, other potential uncertainty model methods can also be used here. DUAL is based on Gaussian processes and it aims to provide predictive uncertainty estimations. At the same time, DUAL can be adapted widely in current industrial recommendation models by maintaining the flexibility of deep neural networks and no extra requirement of the architecture of the deep neural networks. DUAL can be easily implemented on existing models and deployed in real-time industrial recommendation systems.

The loss function of DUAL take both the relevance and the uncertainty into the consideration simultaneously, thus can be written as

$$\mathcal { L } _ { U E _ { s 2 t } } = \mathcal { D } ( f _ { s 2 t }, y _ { s 2 t } ) & & ( 7 ) & & 2.$$

where L 𝑈𝐸 𝑠 2 𝑡 is the loss of 𝑈𝐸 𝑠 2 𝑡 module for source-to-target pair, noted as 𝑠 and 𝑡 respectively, D is the DUAL module, 𝑓 𝑠 2 𝑡 is the logit function of similarity between source and target, 𝑦 𝑠 2 𝑡 is the label that whether source and target are similar.

2.6.2 Modeling Uncertainty of Item-to-Item. For the preparation of item-to-item similarity of training samples, UICR employs the item-to-item matching algorithm to select 𝑆𝑎𝑚𝑝𝑙𝑒 𝑝𝑜𝑠 items from the whole candidate item set with high similarity as positive samples. There are no special requirements in theory for selecting a specific item-to-item matching algorithm to train the 𝑈𝐸 𝑖 2 𝑖 module of the UICR model. Classical item-to-item matching algorithms, like item-based Collaborative filtering, Swing [24], Enhanced Graph Embedding with Side Information (EGES) [21], Node2vec[8], can be used for this purpose. For simplicity, we have chosen the Swing algorithm, which is a widely used and classical item-to-item matching algorithm in the e-commerce domain. Within each training batch, UICR randomly sample 𝑆𝑎𝑚𝑝𝑙𝑒 𝑛𝑒𝑔 items from other instances as negative samples, forming the positive and negative sample sets for the items in the current instance. For each instance, UICR query the original feature data of the samples and use the item tower to perform forward inference, obtaining the corresponding item embeddings. Subsequently, UICR calculates the inner product between the current item embedding and the positive, negative samples respectively, multiply the inner product by the temperature coefficient, then feed the model's output to the 𝑈𝐸 𝑖 2 𝑖 network. The loss function of the 𝑈𝐸 𝑖 2 𝑖 network is noted as L 𝑖 2 𝑖 .

According to the method described above, the item-to-item similarity and variance can be calculated as follow:

$$< s c o r e _ { j, k }, v a r _ { j, k } > = U E _ { i 2 i } ( \tau e _ { j } ^ { T } \mathbf e _ { k } ) \quad \quad ( 8 )$$

where 𝑠𝑐𝑜𝑟𝑒 𝑗,𝑘 is the similarity score between item 𝑗 and item 𝑘 , 𝑣𝑎𝑟 𝑗,𝑘 is the variance of similarity score between item 𝑗 and item 𝑘 , 𝜏 is the temperature coefficient.

2.6.3 Modeling Uncertainty of User-to-Item. Sample preparation for user-to-item uncertainty modeling is much easier. Because the training samples have positive and negative samples generated from multiple possible methods, like user click positive samples, negative sampling samples sampled by different sampling strategies. Generally speaking, the user-to-item uncertainty module can be formalized in equation (9). The output loss of the network is L 𝑢 𝑖 2 . Due to there exists no interaction between item tower input feature and user part of network. Equation (9) can be re-written as

$$< s c o r e _ { u _ { j }, k }, v a r _ { u _ { j }, k } > = U E _ { u 2 i } ( N N _ { u s e r } ( x _ { c }, x _ { u _ { j } }, \mathbf e _ { k } ) ) \quad ( 9 ) \quad \underset { \text{nu} } { \ e p p }.$$

2.6.4 Model Training. In summary, we have completed the introduction to the various components of the model. The loss function for model training can be represented by the following equation, L = D( 𝑓 𝑢 𝑖 2 , 𝑦 𝑢 𝑖 2 ) + 𝜆 𝑖 2 𝑖 D( 𝑓 𝑖 2 𝑖 , 𝑦 𝑖 2 𝑖 ) (10)

where 𝜆 𝑖 2 𝑖 is a hyper parameter which balance the user-to-item and item-to-item losses in the model optimization process. The UICR model is trained using the Adam optimizer. It should be noted that due to the presence of the item-to-item uncertainty estimation network, there may be issues of data leakage and data crossing at multiple stages, including the item-to-item samples preparation in each train batch, the feature queried for the item tower and the order in which training data enters the network. It is crucial to handle these issues carefully to avoid data leakage and data crossing.

2.6.5 Online Deployment. The engineering team use Tensor-RT as the inference engine and enable various optimizations, including but not limited to multi-streaming, precomputing item embeddings offline and making them directly accessed online, FP 32 to 16. With the above optimizations, a complete UICR retrieval process takes approximately an average of 25 𝑚𝑠 . In terms of machine performance, currently, we use GPU (A30), which can achieve several hundreds QPS per GPU.

## 3 EXPERIMENTS

In this section, extensive offline and online experiments are performed on both the large-scale recommender system in Shopee and public benchmark datasets to answer the following research questions:

- · RQ1 Does our proposed method outperform the baseline methods?
- · RQ2 How does each part of our UICR model work?
- · RQ3 How does the model perform when deployed online?

Before presenting the evaluation results, we first introduce the experimental setup, including datasets, baselines, metrics, and parameter settings.

## 3.1 Experimental Setup

3.1.1 Datasets. We adopt public datasets and industrial datasets to comprehensively compare UICR models and baseline models. The statistics of the datasets are shown in Table 1.

Table 1: Statistics of public and industrial datasets.

| Datasets   | Users                                  | Items                                  | Samples                                | Density                                      |
|------------|----------------------------------------|----------------------------------------|----------------------------------------|----------------------------------------------|
| A T Shopee | 2 . 1 × 10 7 9 . 8 × 10 5 4 . 6 × 10 6 | 9 . 4 × 10 6 4 . 2 × 10 6 6 . 3 × 10 5 | 8 . 3 × 10 7 1 . 0 × 10 8 8 . 5 × 10 7 | 4 . 2 × 10 - 7 2 . 4 × 10 - 5 2 . 9 × 10 - 5 |

A dataset 1 : To validate the real-world effectiveness on largescale industrial datasets, we utilize the complete dataset from Amazon, encompassing data from all categories including Books, Movies &amp; TV, Clothing, and others, while other papers that select only a few pure categories. We generate multiple samples by sorting the historical rating sequences of users by time. In other words, we use 𝑏 1 ...𝑏 𝑛 -1 as user features and 𝑏 𝑛 as the target item.

Tdataset 2 : This dataset was first released by Alibaba Taobao and is widely used as a common benchmark in collaborative filtering approaches. It is a user behavior log for Taobao, including click, purchase, add-to-cart, and favorite behaviors.

Industrial Shopee dataset : This dataset is an industrial dataset collected by Shopee App which is one of the top-tier mobile Apps in Southeast Asia. Shopee is also a typical e-commerce scenario, with similar data density between the two.

3.1.2 Baselines. In the industry, the mainstream development paths for matching algorithms can be divided into three categories: 1) collaborative filtering based item-to-item algorithms, 2) vector model based retrieval algorithms, and 3) full-database retrieval algorithms

1 https://tianchi.aliyun.com/dataset/649

2 https://cseweb.ucsd.edu/~jmcauley/datasets.html#amazon\_reviews

based on complex DNN models (such as NANN[2]). From a perspective of technological advancement, it is generally believed that 3) &gt; 2) &gt; 1). Therefore, we selected the recently industry iteration achievement NANN as our main baseline. For classical vectorbased retrieval algorithms, we chose the classic YoutubeDNN and DSSM. Recent iterations have been based on them for multi-interest, multi-objective, and long-term interest iterations. However, these algorithm iteration achievements are extensions based on vector matching models. Because they can not introduce complex intersection architecture, like multi-head attention, to model user historical behavior sequence and candidate item.

We compare the UICR with classical matching methods and recent proposed methods as follows:

- · DSSM DSSM[10] is originated from Natural Language Processing and it is one of the most widely used deep matching model in industial recommendation system.
- · YoutubeDNN YoutubeDNN[3] is one of the most successful deep learning based matching model widely used in industrial recommendation system.
- · NANN NANN upgrades the vector dot product during the retrieval process to a DNN model. This surpasses the upper limit of expressive power in vector models, resulting in significant improvements in the capabilities of the model.

3.1.3 Metrics. In our experiments, we use three metrics to evaluate the performance of the UICR model: 𝑅𝑒𝑐𝑎𝑙𝑙 @ , 𝑁 𝐶𝑎𝑡𝑒𝐸𝑛𝑡𝑟𝑜𝑝𝑦 @ 𝑁 and 𝑁𝑒𝑤𝐶𝑎𝑡𝑒𝑅𝑎𝑡𝑖𝑜 @ 𝑁 [20]. The first metric primarily assess the relevance of the matching results, while the latter two metrics primarily evaluate the novelty of the matching results.The calculation methods are as follows:

$$\text{Recall} \varrho N = | P _ { u } \cap G _ { u } | / | G _ { u } | \text{ \quad \ \ } ( 1 1 ) \text{ \quad \ \ } _ { \text{ni} } ^ { \cdots }$$

where 𝑃 𝑢 ( | 𝑃 𝑢 | = 𝑁 ) denotes the set of retrieved items and 𝐺 𝑢 denotes the set of ground truths. The higher the 𝑅𝑒𝑐𝑎𝑙𝑙 @ , the 𝑁 better the relevance of the model's matching results.

The metric 𝑁𝑒𝑤𝐶𝑎𝑡𝑒𝑅𝑎𝑡𝑖𝑜 is defined as follow:

$$\text{NewCateRatio} \theta N = | P _ { u _ { - } } C a t e \, \langle H _ { u _ { - } } C a t e | / | H _ { u _ { - } } C a t e | \quad ( 1 2 ) \quad \text{elty}$$

where 𝑃 𝑢 \_ 𝐶𝑎𝑡𝑒 stands for the set of categories of items in the retrieved top 𝑁 result, 𝐻 𝑢 \_ 𝐶𝑎𝑡𝑒 means the set of categories of items in the user's historical behavior list, like user click sequence. Therefore, the larger the result computed by equation (12), the better the novelty of the sequences returned by the matching model.

The metric 𝐶𝑎𝑡𝑒𝐸𝑛𝑡𝑟𝑜𝑝𝑦 can be represented as follow,

$$C a t e E n t r o p y = - \sum _ { i = 1 } ^ { \tilde { K } } P r o b _ { P _ { u _ { - } } C a t e _ { i } } \cdot \log _ { 2 } ( P r o b _ { P _ { u _ { - } } C a t e _ { i } } ) \quad ( 1 3 ) \quad \text{$\frac{an i}{N}$}.$$

where 𝑃𝑟𝑜𝑏 𝑃 𝑢 \_ 𝐶𝑎𝑡𝑒 𝑖 represents the probability distribution calculated by category in the matching results for user u. Equation(13) computes the entropy distribution at the category level. If the entropy is larger, it indicates better diversity in the matching results.

3.1.4 Parameter Settings. Wesetthe architecture of fully connected item tower in UICR to [ 64 48 32 , , ] and the dimension of each feature to 32. The final number of neighbors used in the HNSW index structure employed in this paper is uniformly set to 32.The fully connected layer in UICR, which take the context feature, target attention of user sequence and item candidate and item tower output embedding as input, is set to [ 128 64 1 . The temperature coeffi-, , ] cient is set to 10 0 by grid search. All the models reported in this .

Table 2: Results of different methods on public and industrial datasets.All the results listed in this table are calculated with 100 matching outcomes.

| DataSet    | Algorithm                 | Recall               | Entropy                     | NewCateRatio         |
|------------|---------------------------|----------------------|-----------------------------|----------------------|
| A          | DSSM YoutubeDNN NANN UICR | 0.0185 0.0231 0.0290 | 1.9021 2.0482 1.7261 1.7583 | 0.8538 0.8733 0.8313 |
| A          | DSSM                      | 0.0426               | 3.461                       |                      |
| A          |                           | 0.0305               |                             | 0.8427               |
| A          |                           |                      |                             | 0.8195               |
| T          | YoutubeDNN                | 0.0477               | 3.2613                      | 0.8001               |
| A          | NANN                      | 0.0628               | 2.6481                      | 0.7556               |
| A          | UICR                      | 0.0726               | 2.6693                      | 0.7698               |
| Industrial | DSSM                      | 0.0864               | 1.6716                      | 0.5377               |
| Industrial | YoutubeDNN                | 0.0959               | 1.7298                      | 0.5612               |
| Industrial | NANN                      | 0.1202               | 1.4482                      | 0.4837               |
| Industrial | UICR                      | 0.1394               | 1.4865                      | 0.4850               |

paper are implemented in TensorFlow and are trained by Adam optimizer. The parameter 𝜆 𝑖 2 𝑖 in equation (10) is set to 0 1. .

## 3.2 RQ1: Does our proposed method outperform the baseline methods?

In order to validate the effectiveness of the UICR algorithm, we selected three algorithms as baselines for comparison. The comparative performance of UICR and each baseline model on different datasets is reported in Table 2. We have the following observations: 1) DNN vs Vector: The user-to-item networks of NANN and UICR are both based on complex DNNs, while YoutubeDNN and DSSM are based on vector-based single/dual networks. In terms of expressive capability, DNNs are significantly superior to vectors. The 𝑅𝑒𝑐𝑎𝑙𝑙 @100 data from the Table 2 also shows that NANN/UICR significantly outperform YoutubeDNN and DSSM. 2) NANN vs UICR: In terms of metrics, UICR achieves improvements in both relevance and novelty metrics. This is attributed to the UN-Index and UNRetrieval structures, where the UN-Index structure contributes to the improvement in 𝑅𝑒𝑐𝑎𝑙𝑙 @ , and UN-Retrieval enhances nov𝑁 elty (for specific impacts, please refer to the subsequent ablation analysis). 3) In balancing relevance and novelty in recommendation systems, we believe that it is necessary to enhance novelty metrics while ensuring a certain level of relevance. Relevance determines short-term retention of users in the App, while novelty affects longterm user engagement. The greatest contribution of UICR is its ability to simultaneously consider relevance and novelty, achieving an improvement in novelty without compromising relevance.

More importantly, in the historical improvements of matching algorithms, there might have been a tendency to overlook novelty while focusing on improving relevance. We observe the trends in the experimental results of the three models (DSSM, YouTubeDNN, NANN). The improvement in various metrics for each model may come at the expense of sacrificing novelty, which is not conducive to the long-term ecological development of recommendation system.

## 3.3 RQ2: How does each part of our UICR model work?

We explore the contributions of each module in the UICR to the final results through statistical analysis and ablation experiments.

The analysis in this section is based on user behavior data from the Shopee production environment (the Shopee dataset). This

Table 3: Results of ablation study. All the results listed in this table are calculated with 100 matching outcomes.

| Experiment Group                           |   Recall |   Entropy NewCateRatio |   Entropy NewCateRatio |
|--------------------------------------------|----------|------------------------|------------------------|
| A:UICR w/o 𝑈𝐸 𝑖 2 𝑖 & w/o 𝑈𝐸 𝑢 2 𝑖         |   0.1202 |                 1.4482 |                 0.4837 |
| B:UICR w/o 𝑈𝐸 𝑖 2 𝑖 & w 𝑈𝐸 𝑢 2 𝑖           |   0.1247 |                 1.4904 |                 0.4851 |
| C:UICR w 𝑈𝐸 𝑖 2 𝑖 & w 𝑈𝐸 𝑢 2 𝑖 (Full UICR) |   0.1394 |                 1.4865 |                 0.485  |

approach is taken because conclusions drawn from analysis using real-world data in industrial settings have more reference value and practical significance. The UICR introduces two additional modules compared to the baseline model: the 𝑈𝐸 𝑢 𝑖 2 and 𝑈𝐸 𝑖 2 𝑖 modules. Sec. 2.5 and Sec. 2.6 provide an introduction to the approach, data metrics, discussion and analysis of the ablation experiments.

3.3.1 Effect of UN-Index: the item-to-item Uncertainty Estimation Module. A simple method to test the effectiveness of item-to-item confidence is to bucketize the click counts of items in the training dataset. The click counts are grouped into buckets, where each bucket represents a range of 10 clicks. Then, 100 items and their corresponding embeddings are randomly sampled without replacement from the items with click counts in the range ( 10 20 . Simi-, ] larly, 100 items are randomly sampled without replacement from each of the other buckets. The 𝑈𝐸 𝑖 2 𝑖 is used to estimate the variance between the different groups and the average value of variances is calculated. The results are shown in the left part of Figure 3. We can observe that as the number of training samples increases, the average value of variances of the item-to-item gradually decreases, indicating an increase in confidence in the estimated scores. This suggests two things: 1) Our strategy for constructing item-to-item samples is effective. 2) The UN-Model is able to simultaneously model the item-to-item similarity score and the uncertainty.

Figure 3: Relationship between i2i/u2i uncertainty and number of training samples

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00799v2-figure-eaaac08f-caf9-4bff-9643-fc1178b07069.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00799v2-figure-19191280-0f30-4cb3-bdc9-4766e56f7420.jpg)

We trained a version of the complete UICR model by removing the item-to-item uncertainty modeling module ( 𝑈𝐸 𝑖 2 𝑖 ). The performance of this model is presented in Table 3 (Group B vs Group C ). Based on the data, we can draw the conclusion that When the UN-Index module is removed (Group B ), compared to Full UICR (Group C ), the model's 𝑅𝑒𝑐𝑎𝑙𝑙 @100 decreases, aligning with the training goal of UN-Index to enhance the relevance of the index.

3.3.2 Effect of UN-Retrieval: the user-to-item Uncertainty Estimation Module. Similar to the uncertainty validation method for item-toitem, the method for validating the effectiveness of user-to-item uncertainty involves grouping the click counts of each item in the training dataset. From each group, 10 users are randomly sampled without replacement. The 𝑈𝐸 𝑢 𝑖 2 network is invoked to estimate the uncertainty between the different groups, and the average variance of user-to-item is calculated. The results are shown in the right part of Figure 3. We can observe that as the number of training samples increases, the variance of the user-to-item gradually decreases, indicating an increase in confidence in the estimated scores. This suggests that the model is able to simultaneously model both the estimated scores and their confidence.

In order to evaluate the user-to-item confidence modeling module. We also trained a version of the complete UICR model by removing the user-to-item confidence modeling component ( 𝑈𝐸 𝑢 𝑖 2 ). The performance of this model is presented in Table 3 (Group A vs Group B ). It is worth noting that when we introduced the UNRetrieval, there was a slight increase in the Recall metric. It could be attributed to the addition of the uncertainty estimation module for user-to-item pairs (the Dual algorithm), which results in a slight increase in the number of parameters in the network model structure. Upon reviewing the Dual paper, we found that its inclusion also leads to a slight increase in AUC in CTR task, as the introduction of Dual brings about a slight increase in the model's performance due to the incorporation of additional parameters. As for the noveltyrelated metrics, there was indeed a significant improvement after the addition of UN-Retrieval, indicating exploration of the highvariance portion during the retrieval process.

## 3.4 RQ3: How does the model perform when deployed online?

3.4.1 Online A/B experiment Result. We deployed the UICR in the display advertising system on the Shopee App homepage. The online A/B test experiments showed that the UCIR algorithm resulted in an increase of 4 80% in Revenue, an increase of 2 59% in CTR. . .

## 4 RELATED WORKS

Matching model development. Deep matching algorithms are widely prevalent in the industry, personalized recommendation results are generated by constructing positive and negative samples along with network structures. [3, 9] Moreover, research has revealed that employing a singular user embedding proves challenging in effectively capturing the entirety of a user's interests [1, 12, 19] . Consequently, the paradigm of multi-embedding interest modeling approaches has emerged.

Indexing Methods. As the number of candidate items increases, dual-tower matching model encounters performance bottlenecks. Numerous research endeavors have leveraged tree [27-29] and graph structures [2] for modeling purposes. Deep retrieval (DR) [8] encodes all candidate items with learnable paths and train the item paths along with the deep model to maximize the same objective.

## 5 CONCLUSION

In summary, UICR method makes a significant contribution to the relevance and novelty in the matching field. This method includes UN-Index, UN-Retrieval and UN-Model, introducing uncertaintyrelated information during the matching stage to effectively address the challenge of balancing relevance and novelty. By incorporating uncertainty into the matching framework, it not only enhances the quality of the index (relevance) but also improves the novelty of recommendation systems. Additionally, our proposed method has been thoroughly validated through extensive experimental results.

## 6 ACKNOWLEDGEMENTS

We deeply appreciate Shujie Ma for his helpful suggestions and discussions.

## REFERENCES

- [1] Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. 2020. Controllable multi-interest framework for recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining . 2942-2951.
- [2] Rihan Chen, Bin Liu, Han Zhu, Yaoxuan Wang, Qi Li, Buting Ma, Qingbo Hua, Jun Jiang, Yunlong Xu, Hongbo Deng, et al. 2022. Approximate nearest neighbor search under neural similarity metric for large-scale recommendation. In Proceedings of the 31st ACM International Conference on Information &amp; Knowledge Management . 3013-3022.
- [3] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems . 191-198.
- [4] Chao Du, Zhifeng Gao, Shuo Yuan, Lining Gao, Ziyan Li, Yifan Zeng, Xiaoqiang Zhu, Jian Xu, Kun Gai, and Kuang-Chih Lee. 2021. Exploration in online advertising systems with deep uncertainty-aware learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &amp; Data Mining . 2792-2801.
- [5] Zeshan Fayyaz, Mahsa Ebrahimian, Dina Nawara, Ahmed Ibrahim, and Rasha Kashef. 2020. Recommendation systems: Algorithms, challenges, metrics, and business opportunities. applied sciences 10, 21 (2020), 7748.
- [6] Zhe Fu and Xi Niu. 2023. Modeling Users' Curiosity in Recommender Systems. ACM Transactions on Knowledge Discovery from Data 18, 1 (2023), 1-23.
- [7] Juan Gong, Zhenlin Chen, Chaoyi Ma, Zhuojian Xiao, Haonan Wang, Guoyu Tang, Lin Liu, Sulong Xu, Bo Long, and Yunjiang Jiang. 2023. Attention Weighted Mixture of Experts with Contrastive Learning for Personalized Ranking in Ecommerce. arXiv preprint arXiv:2306.05011 (2023).
- [8] Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining . 855-864.
- [9] Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embeddingbased retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining . 2553-2561.
- [10] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information &amp; Knowledge Management . 2333-2338.
- [11] Shristi Shakya Khanal, PWC Prasad, Abeer Alsadoon, and Angelika Maag. 2020. A systematic review: machine learning based recommendation systems for elearning. Education and Information Technologies 25 (2020), 2635-2664.
- [12] Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. 2019. Multi-interest network with dynamic routing for recommendation at Tmall. In Proceedings of the 28th ACM international conference on information and knowledge management . 2615-2623.
- [13] Wanda Li, Wenhao Zheng, Xuanji Xiao, and Suhang Wang. 2023. STAN: StageAdaptive Network for Multi-Task Recommendation by Learning User LifecycleBased Representation. arXiv preprint arXiv:2306.12232 (2023).
- [14] Kachun Lo and Tsukasa Ishigaki. 2021. PPNW: personalized pairwise novelty loss weighting for novel recommendation. Knowledge and Information Systems 63 (2021), 1117-1148.
- [15] Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824-836.
- [16] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2020. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 (2020).
- [17] Arseto Satriyo Nugroho, Igi Ardiyanto, and Teguh Bharata Adji. [n. d.]. User Curiosity Factor in Determining Serendipity of Recommender System. IJITEE (International Journal of Information Technology and Electrical Engineering) 5, 3 ([n. d.]), 75-81.
- [18] Dvir Samuel and Gal Chechik. 2021. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 9495-9504.
- [19] Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, and Xia Hu. 2021. Sparse-interest network for sequential recommendation. In Proceedings of the 14th ACM international conference on web search and data mining . 598-606.
- [20] Saúl Vargas and Pablo Castells. 2011. Rank and relevance in novelty and diversity metrics for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems . 109-116.
- [21] Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. 2018. Billion-scale commodity embedding for e-commerce recommendation in alibaba. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery &amp; data mining . 839-848.
- [22] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. 2020. Longtailed recognition by routing diverse distribution-aware experts. arXiv preprint
- arXiv:2010.01809 (2020).
- [23] Jun Xu, Xiangnan He, and Hang Li. 2018. Deep learning for matching in search and recommendation. In The 41st International ACM SIGIR Conference on Research &amp; Development in Information Retrieval . 1365-1368.
- [24] Xiaoyong Yang, Yadong Zhu, Yi Zhang, Xiaobo Wang, and Quan Yuan. 2020. Large scale product graph construction for recommendation in e-commerce. arXiv preprint arXiv:2010.05525 (2020).
- [25] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. 2023. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
- [26] Pengfei Zhao and Dik Lun Lee. 2016. How much novelty is relevant? it depends on your curiosity. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval . 315-324.
- [27] Han Zhu, Daqing Chang, Ziru Xu, Pengye Zhang, Xiang Li, Jie He, Han Li, Jian Xu, and Kun Gai. 2019. Joint optimization of tree-based index and deep model for recommender systems. Advances in Neural Information Processing Systems 32 (2019).
- [28] Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. 2018. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining . 1079-1088.
- [29] Jingwei Zhuo, Ziru Xu, Wei Dai, Han Zhu, Han Li, Jian Xu, and Kun Gai. 2020. Learning optimal tree models under beam search. In International Conference on Machine Learning . PMLR, 11650-11659.
Xin Jiang Kaiqiang Wang Yinlong Wang Fengchang Lv Taiyang Peng Shuai Yang Xianteng Wu Pengye Zhang Shuo Yuan Yifan Zeng
Deep Uncertainty-Based Explore for Index Construction and Retrieval in Recommendation System
In recommendation systems, the relevance and novelty of the final results are
selected through a cascade system of Matching -> Ranking -> Strategy. The
matching model serves as the starting point of the pipeline and determines the
upper bound of the subsequent stages. Balancing the relevance and novelty of
matching results is a crucial step in the design and optimization of
recommendation systems, contributing significantly to improving recommendation
quality. However, the typical matching algorithms have not simultaneously
addressed the relevance and novelty perfectly. One main reason is that deep
matching algorithms exhibit significant uncertainty when estimating items in
the long tail (e.g., due to insufficient training samples) items.The
uncertainty not only affects the training of the models but also influences the
confidence in the index construction and beam search retrieval process of these
models. This paper proposes the UICR (Uncertainty-based explore for Index
Construction and Retrieval) algorithm, which introduces the concept of
uncertainty modeling in the matching stage and achieves multi-task modeling of
model uncertainty and index uncertainty. The final matching results are
obtained by combining the relevance score and uncertainty score infered by the
model. Experimental results demonstrate that the UICR improves novelty without
sacrificing relevance on realworld industrial productive environments and
multiple open-source datasets. Remarkably, online A/B test results of display
advertising in Shopee demonstrates the effectiveness of the proposed algorithm.