Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search

Sequential recommender systems (SRS) have become a research hotspot due to its power in modeling user dynamic interests and sequential behavioral patterns. To maximize model expressive ability, a default choice is to apply a larger and deeper network architecture, which, however, often brings high network latency when generating online recommendations. Naturally, we argue that compressing the heavy recommendation models into middle- or light- weight neural networks is of great importance for practical production systems. To realize such a goal, we propose AdaRec, a knowledge distillation (KD) framework which compresses knowledge of a teacher model into a student model adaptively according to its recommendation scene by using differentiable Neural Architecture Search (NAS). Specifically, we introduce a target-oriented distillation loss to guide the structure search process for finding the student network architecture, and a cost-sensitive loss as constraints for model size, which achieves a superior trade-off between recommendation effectiveness and efficiency. In addition, we leverage Earth Mover's Distance (EMD) to realize many-to-many layer mapping during knowledge distillation, which enables each intermediate student layer to learn from other intermediate teacher layers adaptively. Extensive experiments on real-world recommendation datasets demonstrate that our model achieves competitive or better accuracy with notable inference speedup comparing to strong counterparts, while discovering diverse neural architectures for sequential recommender models under different recommendation scenes.

INTRODUCTION
S EQUENTIAL (a.k.a. session-based) recommender systems that aim to predict new interactions based on user historical ones have attracted much attention in recent years. Particularly, with the tremendous success of deep learning, deep neural network (DNN) based sequential recommendation (SR) models have yielded substantial improvements comparing to traditional collaborative filtering (CF) [1], such as neighborhood methods [2] and shallow factorization models [3]. This is because with many hidden layers, well-designed deep models could be more powerful in capturing user dynamic interests, high-level or long-range sequential relations of user interactions. More recently, Wang et al. [4] and Chen et al. [5] revealed that deep SR models such as NextItNet [6] and SASRec [7] could be stacked in a surprised depth with over 100 layers for achieving their optimal performance. However, a real problem arises as these deep SR models go bigger and deeper; that is, the model becomes too large in parameter size, and both memory and inference costs increase sharply, making the deployment of them difficult in production systems. Thereby, we argue that compressing the heavy deep SR models into moderate-or light-weight neural networks without Lei  sacrificing their accuracy is of crucial importance for practical usage. Knowledge Distillation (KD) [8] as an effective compression technique has been recently investigated in the recommender systems domain [9], [10]. By transferring useful knowledge from a big teacher network to the student network, large deep models could be slimmed into a smaller and shallower structure without performance degradation. However, existing KD methods basically distill the teacher model into a fixed-structure student model that is manually designed in advance. This potentially limits the flexibility and scalability of the student model, especially for diverse and relatively complicated scenarios in recommender systems. For example, the optimal structure for music recommendation might be different from the optimal structure for E-commerce recommendation. Ideally, we hope to generate an adaptive student model whose optimal structure considers the specific recommendation scenarios. Inspired by the success of automated machine learning (Au-toML), we propose a novel knowledge distillation method to compress the deep sequential recommendation models, termed AdaRec. AdaRec distills the knowledge of a teacher model into a student model adaptively according to the recommendation scene based on differentiable Neural Architecture Search (NAS) [11], [12], [13]. Specifically, we devise a target-oriented knowledge distillation loss to provide search supervision for learning the architecture of student network, and a cost-sensitive loss as additional regularization to constrain the model size, which achieve a superior trade-off between recommendation effectiveness and efficiency. In addition, we leverage Earth Mover's Distance (EMD) to realize effective many-to-many layer mapping during the distillation process, enabling each intermediate layer of student to learn from any other intermediate layers of its teacher. It is worth noting arXiv:2107.07173v2 [cs.IR] 10 Apr 2022 that, our method is a generic knowledge distillation framework which can directly apply to a broad class of well-known sequential recommendation models, such as NextItNet [6], SASRec [7] and BERT4Rec [14]. Besides, with the well-designed NAS architecture, our method can distill the deep sequential recommendation models into effective smaller models with diverse neural network architectures, according to the specific recommendation scenarios.
Our contributions in this paper are fourfold: • To the best of our knowledge, we are the first to propose combining Knowledge Distillation and Neural Architecture Search in the SRS tasks so as to adaptively compress the deep sequential recommendation models according to recommendation scenes. • We devise a knowledge distillation loss based on Earth Mover's Distance (EMD) and a cost-sensitive constraint to achieve a trade-off between recommendation effectiveness and efficiency. • We verify the universality of the AdaRec framework by performing KD with three different teacher models, namely, NextItNet [6], SASRec [7] and BERT4Rec [14]. • We conduct extensive experiments on three real-world recommendation datasets with different scenarios, demonstrating that AdaRec achieves competitive or better accuracy with notable inference speedup comparing to its original teacher model. Moreover, we discover diverse neural architectures of the student model in different recommendation scenarios or tasks.

Deep Sequential Recommendation
Sequential (a.k.a. session-based) recommender systems (SRS) is an important branch in the recommendation field and has become a hotspot recently due to the wide range of application scenarios and huge commercial values. Since in this paper we focus on compressing large and deep sequential recommendation models, we only review related work regarding its advancement in deep learning (DL). Deep neural networks have achieved superior recommendation accuracy in SRS tasks. In general, these models could be classified into three categories, namely RNN-based, CNN-based and self-attention based methods. Specifically, Hidasi et al. [1] proposed GRU4Rec, which is the first RNN-based sequential recommendation model. Following this work, many extended works were proposed, which either optimized a new ranking loss [15], incorporated more context features [16], or developed more advanced data augmentation [17]. While effective, these models rely heavily on the hidden states of the entire past, which cannot take full advantage of the parallel processing resources (e.g., GPU and TPU) [6] during training. Therefore, Convolutional Neural Network (CNN) and self-attention based models are proposed to mitigate such limitations [6], [7], [14], [18]. Among them, Tang et al. [18] proposed Caser, which embeds a sequence of useritem interactions into an "image" and learn sequential patterns as local features of the image by using wide convolutional filters. Subsequently, [6] proposed NextItNet, a very deep 1D temporal CNN-based recommendation model which particularly excels at modeling long-range item sequences. In addition, self-attention based models, such as SASRec [7] and BERT4Rec [14], also showed competitive accuracy for SRS tasks as well. SASRec [7] utilized the popular self-attention mechanism to model longterm sequential semantics by encoding user's historical behaviors. Inspired by the great success of BERT [19] in NLP filed, Sun et al. [14] proposed BERT4Rec, which uses the transformer architecture and masked language model to learn bidirectional item dependencies for better sequential recommendations. In this paper, we present AdaRec by applying NextItNet, SASRec & BERT4Rec as teacher networks given their superior performance in literature. In addition, Wu et al. [20] proposed SSE-PT, a personalized transformer model which applies stochastic shared embeddings (SSE) regularization to achieve personalized user representations. With the advancement on Graph Neural Networks (GNN), GNN-based sequential models, such as SR-GNN [21], GC-SAN [22] and MA-SAN [23], have also attracted attention yielded substantial improvements in recommendation accuracy. Besides, there are some other works [24], [25], [26] that designs novel neural network modules for sequential recommendations. For instance, HGN [24], a hierarchical gating neural network, adopts a feature gating and an instance gating to determine what item features should be used for recommendation. SDM [25] integrates a multi-head self-attention module with a gated fusion module to capture both short-and long-term user preferences. HAM [26] develops hybrid associations models to further capture sequential and multi-order user-item association patterns for sequential recommendations.

Knowledge Distillation
Large and deep neural networks have achieved remarkable success in recent recommendation literature [4], [27], [28]. However, the deployment of such heavy model for real production system remains a great challenge. Knowledge Distillation (KD) [8], [29] is a representative technique for model compression and acceleration. Its basic idea is to transfer important knowledge from a big teacher network to a small student network. Specifically, Tang et al. [9] proposed the first KD technique for learning to rank problems in recommender systems. However, the work only focused on distillation on very shallow neural recommendation models while its effectiveness for deep SRS keeps largely unknown. [30] presented a general knowledge distillation framework for counterfactual recommendation with four types of distillation, namely, labelbased, feature-based, sample-based and model structure-based distillation. More recently, [31] proposed a knowledge distillation framework that forces the student network to learn from both the teacher's output and the latent knowledge stored in the teacher model.
KD-based compression have also been widely studied in other domains [28], [29], [32], [33]. Nowak et al. [34] proposed a structure compression method which involves transferring the knowledge learned by multiple layers to a single layer. Wang et al. [35] progressively performed block-wise knowledge transfer from teacher networks to student networks while preserving the receptive field. Mirzadeh et al. [36] introduced a teacher assistant to mitigate the training gap between teacher model and student model. Recently, compressing pretrained language models (e.g., BERT) with KD has attracted increasing attention, and numerous novel models are proposed to effectively distill BERT from different perspectives (e.g., embedding layer, hidden layers and prediction layer), such as PKD-BERT [32], DistilBERT [28], TinyBERT [33] and BERT-EMD [29].    The proposed AdaRec consists of two primary components: teacher model and student model. In this paper, we specify AdaRec using NextItNet [6], SASRec [7] and BERT4Rec [14] as the teacher models given their superior recommendation performance. The structures of the student models are searched based on neural architecture search techniques in a differentiable manner. Specifically, we devise a target-oriented knowledge distillation loss to provide search hints for searching the architecture of student network, and an efficiency-aware loss as search constraints for constraining the model size, which achieves a superior trade-off between effectiveness and efficiency for sequential recommendations.

Neural Architecture Search
Neural Architecture Search (NAS) that automatically discovers the network architecture, has gained increasing attention recently. Early NAS methods based on reinforcement learning [37] and evolution [38] are computationally very expensive. Recent studies significantly speed up the search and evaluation stages by architecture parameter sharing, such as ENAS [39], gradient-descent based DARTS [11], [13] and SNAS [12], and hardware-aware optimization such as AMC [40] and FBNet [41], [42]. Different from existing work, we devise a target-oriented knowledge distillation loss to provide search supervision for learning the architecture of the student network, which is a joint search of student structure and knowledge transfer under the guidance of the teacher model. To our best knowledge, we are the first to propose a combination of KD and NAS for compressing the deep sequential recommendation models.

PROBLEM DEFINITION
Given a sequence of user's historical behaviors denotes the t-th interacted item of user u, the goal of SRS is to infer the item x u t+1 that the user would like to interact with at time t + 1. Since users usually pay attention to only the first few items, the top-N items are recommended, referred to as the top-N item recommendation problem.
Deep neural networks, such as NextItNet [6], SASRec [7], BERT4Rec [14], have been proposed and deployed to many SRS applications, yielding state-of-the-art performances. However, different from many traditional collaborative filtering models [43], deep sequential recommendation models often require more hidden layers to model complex and long-term relations of user actions. Recent work in [27] showed that the state-of-the-art temporal CNN model NextItNet needs over 30 layers to reach its maximum expressive ability on some benchmark datasets. Even more, [4], [5] demonstrated by experiments that both NextItNet and SASRec [7] should be stacked with over 100 layers for achieving their best accuracy. This could lead to a large model size and high network latency in practice, bringing difficulties for the deployment of them in production systems. Therefore, in this paper, we hope to reduce the model size and accelerate the inference speed for these very deep sequential recommendation models without sacrificing their accuracy.

ADAREC
We introduce a novel scene-adaptive KD-based model compression approach with differentiable NAS, called AdaRec. Formally, suppose that a large teacher model T is trained on a target dataset D, and the architecture searching space is denoted as A. The goal of AdaRec is to automatically find a high-performing student model S from A with a smaller scale. Figure 1 illustrates the overview of the AdaRec framework. The basic idea is to transfer knowledge from a large teacher recommender model T to a small student model S adaptively subject to the specific recommendation task. In this paper, we specify AdaRec using NextItNet [6], SASRec [7] and BERT4Rec [14] learning algorithm as the teacher models given their superior recommendation performance. It is noteworthy that the "teacher" is model-agnostic and potentially applicable for any sequential recommendation model with a deep network architecture. Specifically, the network structures of the student model are automatically searched based on the NAS techniques. To this end, we devise a KD loss to provide search supervision for learning the architecture of the student network and a cost-sensitive loss as search regularization to control the model size. In this manner, our AdaRec could achieve a superior trade-off between effectiveness and efficiency for SRS tasks.
In what follows, we describe AdaRec by elaborating its teacher model, student model, the KD process and the NAS searching process.

Teacher Model
We employ the block-wise (e.g., ResNet [44]) deep networks as the teacher models given their powerful performance in literature. The general framework of the teacher model consists of the bottom embedding layer, hidden layers and the softmax layer.
In terms of the hidden layers, we use the residual blocks from NextItNet [6], SASRec [7] and BERT4Rec [14] for case study, where NextItNet is based on the dilated CNN blocks, SASRec and BERT4Rec are based on the self-attention blocks. The residual block structures are depicted in Figure 1.
4.1.0.1 NextItNet: NextItNet is composed of a stack of dilated convolutional (DC) layers, which are wrapped by a residual block structure every two layers. Specifically, each input item x u is converted into an embedding vector e u , and the user-item interaction sequence X u is thereby represented by an The embedding sequence E u is then passed into a stack of dilated convolutional layers to learn feature vector E u l which is expected to capture the longrange dependencies. Here, l represents the l-th residual block and each residual block connects two consecutive DC layers. Formally, the l-th residual block with the DC operation is formalized as: where E u l−1 and E u l are input and output of the l-th residual block considered. λ × F l (E u l−1 ) + E u l−1 is a shortcut connection by element-wise addition. Similar to [4], [5], [45], we add a learnable coefficient λ to the residual mappings F l (E u l−1 ), so that the model can stack more layers, and get better results than the standard version with λ as 1. 1 F l (E u l−1 ) represents the residual mapping, which is defined as: where ψ 1 and ψ 2 represent the casual convolution operations. LN 1 and LN 2 represent layer normalization functions. σ is the ReLU activation function. Finally, a softmax output layer is applied to predict the probability distribution for the next item x u t+1 : where W is a projection matrix, and b is a bias term. 4.1.0.2 SASRec: Similar to NextItNet, SASRec is composed of a stack of self-attention (SA) layers, which are wrapped by a residual block with a self-attention layer and a feed-forward network. Formally, the l-th residual block with the SA operation is formalized as: where E u l−1 and E u l are input and output of the l-th residual block considered. λ × H l (E u l−1 ) + E u l−1 is a shortcut connection by element-wise addition. As mentioned above, We also add a learnable coefficient λ to the residual mappings H l (E u l−1 ). H l (E u l−1 ) represents the residual mapping, which is defined as: where FFN and SA represent the feed-forward and self-attention operation, respectively. LN 1 and LN 2 represent layer normalization functions. δ is the dropout function. Finally, a softmax output layer is applied to predict the probability distribution for the next item x u t+1 . For both NextItNet+ and SASRec+, the joint probability p (X u ; Ω) of each user-item interaction sequence is computed by the product of conditional distributions over interacted items as follows: where p x u i |x u 1:i−1 ; Ω is the predicted probability for the i-th item x u i conditioned on all its previous interactions [x u 1 , . . . , x u i−1 ], and Ω is the set of parameters. 4.1.0.3 BERT4Rec: BERT [19] has shown superior performance in many NLP tasks and the recently proposed BERT4Rec [14] has successfully applied the bidirectional transformer structure to SRS tasks, which achieves state-of-the-art performance on sequential recommendations. Compared to SAS-Rec using a unidirectional (left-to-right) transformer structure to capture user's dynamic interests, BERT4Rec takes bidirectional dependencies of user's sequential behaviors into consideration, and proposes a novel Masked Language Model objective to predict the masked items in the interaction sequence. Akin to SASRec, the hidden representations of l-th layer in BERT4Rec are defined similarly as Eq. (4) and (5).
During training, BERT4Rec allows m interactions in the sequence (termed as x ∆ = [x ∆1 , . . . , x ∆m ]) to be masked (i.e., replaced with a special token "[mask]") and the original interaction sequence X u is modified toX u . The goal of the Masked Language Model objective is to predict the original ids of the masked items based solely on its left and right context, which can be formalized as: where Θ is the set of parameters.
Obviously, there is a mismatch between the training and the inference since the Masked Language Model objective aims to predict the current masked items while SRS task aims to predict the next item in the future. To address this, BERT4Rec propose to append the special token "[mask]" to the end of user's behavior interaction sequence, and then predict the next item based on the final hidden representation of this token.

Student Model
Typical model compression methods usually apply KD to transfer knowledge from the large teacher model to the manually designed student model, which rely heavily on the prior knowledge of human experts to design the structure of the student model. We perform architecture search of the student network using NAS techniques rather than assigning a fixed structure in advance. Here, we introduce a block-based micro search strategy [39] to find an optimal network architecture from the search space formed by the operation sets. The search space design is key to the final performance of the searched student model. In this study, we modularize the large search space of NAS into blocks to reduce its complexity, similar to [47]. In this way, one merely needs to search a few block structures and then repeatedly stack such blocks to form the final network architecture. This strategy avoids training each block from scratch, but forces all blocks to share structures, thereby greatly reducing the time to obtain the best performing student model from a large number of candidate networks. Specifically, the searched block denoted by α c is represented as a directed acyclic graph (DAG). Each node of the block indicates a latent state h and the edge from node i to j indicates operation o i,j that transforms h i to h j . For the k-th (k > 1) searched block, we define an input node c k−1 , and an output node c k that is obtained by attentively summarized over all intermediate nodes. Formally, let O be the set of candidate operations, and we assume there is a topological order among M intermediate nodes, i.e., o i,j ∈ O exists when i < j and j ≥ 1, the search space A is thus formalized as: 4.2.0.2 Operation Set: For all the three teacher models (NextItNet, SASRec, BERT4Rec), we adopt the same operation set to search the student network architecture. In this paper, we employ lightweight CNN-based operations as candidates given that they have shown both competitive accuracy and superior efficiency in the SRS literature, compared to RNN [1] and selfattention [7] based models. The candidate operation set O is composed of four kinds of operations: convolution, pooling, skip connection and zero operation. The convolution operations include the 1D convolution, standard convolutions (without dilation), casual dilated convolutions [48] with kernel size {3, 5}. Note that the dilated convolution is used to capture long-term dependency information. The pooling operations include max pooling and average pooling with kernel size 3. The skip operation is leveraged to construct residual connections. The zero operation helps to forget the past knowledge.

Knowledge Distillation
Specifically, we distill knowledge from the teacher model from three perspectives: the embedding layer, the hidden layers, and the prediction layer. 4.3.0.1 Embedding Layer Distillation: The prediction accuracy of the sequential recommendation model, such as Nex-tItNet, can be largely improved by increasing the embedding dimension [27]. Compressing item embedding matrices without sacrificing model performance is essential online inference speedup and parameter reduction. We define L emb as the distillation loss of the embedding layer, where it is minimized by the mean squared error (MSE) between the teacher model and the student model: where E T and E S represent the item embedding matrices of teacher and student models, respectively. W e is a learnable projection parameter. 4.3.0.2 Prediction Layers Distillation: The student model is encouraged to match the prediction ability of the teacher model by learning from the probability logits of the teacher model. We define L pred using KL divergence [49] as the distillation loss of the prediction layer: where z T and z S are probability logits after passing through the softmax layer of the teacher & student models, respectively. 4.3.0.3 Hidden Layers Distillation: Since there are different numbers of hidden layers in teacher and student, conventional one-to-one layer mapping algorithms cannot be applied. Here, we employ the Earth Mover's Distance (EMD) [50] algorithm to encourage each student hidden layer to learn from multiple teacher layers adaptively. EMD measures the distance between the teacher and student networks as the minimum cumulative cost of knowledge transfer [50].
The key insight is to view network layers as distributions, and the desired transformation should make the two distributions (i.e., teacher and student layers) close.
subject to the following constraints: After solving the above optimization problem, we obtain the optimal mapping flow F H . The Earth Mover's Distance (EMD) can be defined as the work normalized by the total flow: Finally, the hidden-layer distillation loss (termed as L hidden ) can be defined by the EMD between H T and H S : By combining the above three distillation objectives (L emb , L pred , L hidden ), we can unify the knowledge distillation loss L KD between the teacher model and the student model: model into the main objective to achieve a trade-off between recommendation effectiveness and efficiency. Specifically, we define a cost-sensitive loss by considering both the parameter size and inference time: where SIZE(·) and F LOP s(·) are the normalized parameter size and the number of floating point operations (FLOPs) for each operation. The sum of FLOPs of searched operations can be used to approximate the actual inference time of the student model.

Training Procedure
Follow the common practice, we first pre-train the teacher model and then search for the student structure under the supervision of the pre-trained teacher model. When searching the student architecture, we combine the knowledge distillation loss L KD and the cost-sensitive loss L E . Besides, we also need to incorporate the cross-entropy loss (L CE ) w.r.t ground-truth labels from the training data to assist the searching process, which is defined as: where X represents the whole user-item interaction sequences in the training data, p(x u t+1 ) is the ground truth distribution for next item prediction and p(x u t+1 ) is the prediction distribution of the searched student model.
The overall loss function is defined as follows: where γ & β are hyper-parameters that balance these loss functions. After finishing the joint search of student structure and knowledge transfer under the guidance of the pretrained teacher model, we can derive an effective, efficient and adaptive architecture as the compressed sequential model by stacking the searched block structures where g i is a random noise drawn from Gumbel(0, 1) distribution, τ is a temperature coefficient to control the discreteness of the output vectors y o . In this way, we can optimize the objectives L KD and L E directly using gradient-based optimizers by using the discrete variable argmax(y o ) in the forward pass and using the continuous vector y o in the back-propagation stage.

Experimental Datasets
We conduct extensive experiments on three real-world SRS datasets from three different domains (scenes): RetailRocket from the E-commerce domain, 30Music from the music domain [52], and MovieLens-2K from the movie domain [53]. The statistics of them are provided in Table 1.
• RetailRocket 2 contains user purchasing and clicking behaviors. We set the maximum length of each sequence t to 10 so as to investigate the recommendation performance with short-range sequences. We split the sequences longer than t into multiple sub-sequences, while the ones shorter than t are padded with zero in the beginning of each sequence, similar to [6]. • 30Music is a collection of listening and playlists data retrieved from Internet radio stations through Last.fm API 3 . We process it as a middle-range sequential dataset by extracting the latest 20 actions per user. • MovieLens-2K 4 (denoted as ML-2K) is a benchmark dataset for both standard collaborative filtering and sequential recom-

Baselines and Evaluation Metrics
To verify the effectiveness and efficiency of AdaRec, we compare it with its teacher model includingNextItNet [6], SASRec [7] and BERT4Rec [14] which have been described in Section 4.1.
In addition, we have also compare it with GRU4Rec [1] and Caser [18] for reference given that the two models are recognized as two most typical sequential recommendation baselines. Following [48], we train Caser using the data augmentation method and train GRU4Rec based on the auto-regressive method. To evaluate recommendation accuracy, we adopt three popular top-N ranking metrics, including MRR@N (Mean Reciprocal Rank), HR@N (Hit Ratio) and and NDCG@N (Normalized Discounted Cumulative Gain) [54]. Here N is set to 5 and 20 for comparison. To evaluate the computational efficiency of AdaRec, we compare its model size and inference speedup with the teacher models.

Implementation Details
We divide the user-item interaction sequence X u = [x u 1:t ] for each user u into X u train = [x u 1:t−2 ] for training, x u t−1 for validation and x u t for testing, following [7]. For the teacher model NextItNet, we set the embedding dimension d to be 256, and use dilation factors of 8 × {1, 2, 4, 8} (32 layers or 16 residual blocks). For both SASRec and BERT4Rec, we set d to be 128 given that a larger d hurts their performance because of overfitting. We use 8 self-attention blocks with four heads for SASRec and BERT4Rec according to its accuracy in the validation set. When searching the architecture for the student model, we set d to one quarter of its teacher's embedding dimension (i.e., d = 64 for NextItNet and d = 32 for SASRec and BERT4Rec), γ = 0.5, β = 8, inner nodes M = 3 and student blocks K = 4. For training AdaRec, we employ AdamW [55] to optimize the parameters (e.g., embedding  Table 2 reports the performance (i.e., MRR@N , HR@N and NDCG@N (N is set to 5 and 20), parameter size and inference speedup) of AdaRec and baseline models on the three datasets.

Overall Results
From the results, we can make the following observations. First, we observe that NextItNet, SASRec and BERT4Rec outperform GRU4Rec and Caser with substantial improvements in terms of recommendation accuracy among the three datasets, which is consistent with the previous work [6], [7], [14]. Second, AdaRec with NextItNet, SASRec and BERT4Rec as teacher models attain competitive or better recommendation accuracy than their teachers, although we do not expect AdaRec beats its teacher model in accuracy. For example, on RetailRocket and 30Music, AdaRec with NextItNet as the teacher model obtains 2.9% and 3.2% improvements over its large teacher model in terms of MRR@5. Importantly, AdaRec requires much fewer parameters and achieves notable inference speedup relative to its teachers. In addition, compared to the standard KD method [8] with equivalent model size, AdaRec with NAS techniques performs substantially better with higher inference speedup.

Cross-Scene Evaluation
In this section, we investigate the scene-adaptivity of AdaRec with different recommendation scenarios. We apply the searched student architecture from one recommendation scenario to other scenarios. For example, we denote the searched student architecture for RetailRocket (i.e., E-commerce domain) with NextItNet as the teacher model as AdaRec-RetailRocket, and apply it to 30Music (i.e., music domain) and ML-2K (i.e., movie domain). For such cross-scenario validation, we randomly initialize the weights of each searched student structure and re-train it using corresponding training data and the same teacher model to ensure a fair comparison. The results are summarized in Table 3, where we omit results using SASRec and BERT4Rec as teacher models due to similar behaviors. As clearly demonstrated along the diagonal line of Table 3, we can draw that AdaRec achieves the best performance on their original recommendation scenarios in contrast to other scenarios. This is, AdaRec is scene-adaptive since the searched student network only guarantees its optimal performance on a specific recommendation scenario.

Architecture Visualization
To better understand the basic blocks of the searched student architectures, we visualize them on the three recommendation scenarios in Figure 2. For space reason, we still only show AdaRec with NextItNet as the teacher model. By comparing the searched structures for different recommendation scenarios, we can find that AdaRec for RetailRocket (from E-commerce domain) and 30Music (from music domain) are relatively lightweight, since fewer convolution operations (i.e., std cnn 3 for RetailRocket and cau cnn 3 for 30Music) are used. This is likely because the two datasets have short-range sequential dependencies. On the contrary, a more complicated student structure with diverse convolution operations(i.e., std cnn 3 and cau cnn 3 ) is learned for ML-2K so as to model the long-range dependencies. The above results well back up our claim that the proposed AdaRec is able to search adaptive student structures for different recommendation scenarios.

Ablation Studies
As described before, the loss L of AdaRec consists of three parts: the target-oriented KD loss L KD , the cost-sensitive loss L E and the standard cross-entropy loss L CE . First, we evaluate the effects of L KD and L CE by removing each of them independently, as reported in Table 4. Clearly, we find that AdaRec without each of the two losses yields sub-optimal recommendation accuracy on all three datasets. Besides, it also shows that combining distillation losses on the embedding layer L emb , prediction layer L pred and hidden layers L hidden together produces the best results. In addition, we verify the effect of the cost-sensitive loss L E by varying β, including the default case β = 8, without constraint β = 0, weak constraint β = 4 and strong constraint β = 16. The model performance and corresponding model size are illustrated in Figure 3. From the results we can see that no constraint or a small value of β lead to an increased model size; meanwhile, an aggressive β results in a smaller model size but degraded model accuracy on the other hand. An appropriate constraint (β = 8) achieves the superior trade-off between the model effectiveness and efficiency.

CONCLUSION
In this paper, we present a novel sequential recommendation knowledge distillation (KD) framework AdaRec based on the differentiable Neural Architecture Search (NAS). AdaRec compresses knowledge of large and deep sequential recommendation models into a compact student model adaptively with the recommendation scene. To the best of our knowledge, AdaRec is the first to propose combining knowledge distillation and neural architecture search in the SRS tasks so as to adaptively compress the deep sequential recommendation models according to different recommendation scenes. Besides, we devise the Earth Mover's Distance (EMD) based KD method for effective transfer of deep hidden layers and introduce a cost-sensitive constraint to achieve the trade-off between effectiveness and efficiency of SRS tasks. Extensive experiments on three real-world recommendation datasets from different scenarios demonstrate that AdaRec achieves considerably better accuracy compared to the standard KD baseline and comparable accuracy with its teacher model while accelerating inference time and reducing the computational workload largely.


Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline

Pre-trained deep language models~(LM) have advanced the state-of-the-art of text retrieval. Rerankers fine-tuned from deep LM estimates candidate relevance based on rich contextualized matching signals. Meanwhile, deep LMs can also be leveraged to improve search index, building retrievers with better recall. One would expect a straightforward combination of both in a pipeline to have additive performance gain. In this paper, we discover otherwise and that popular reranker cannot fully exploit the improved retrieval result. We, therefore, propose a Localized Contrastive Estimation (LCE) for training rerankers and demonstrate it significantly improves deep two-stage models.

Introduction
Recent state-of-the-art retrieval systems are pipelined, consisting of a first-stage heuristic retriever such as BM25 that efficiently produces an initial set of candidate results followed by one or more heavy rerankers that rerank the most promising candidates [11]. Neural language models (LM) such as BERT [7] have had a major impact on this architecture by providing more effective index terms [12] and term weights [5] for heuristic retriever and providing rich contextualized matching signals between query and document for rerankers [4,10].
Intuitively, a better initial ranking provides later stage neural rerankers with more relevant documents to pull up to the top of the final ranking. In a perfect world, a neural reranker recognizes the relevant documents in its candidate pool, inheriting all of the successes of previous retriever. However, simply forming the pipeline by appending a BERT reranker to an effective first-stage retriever does not guarantee an effective final ranking. An improved candidate list sometimes causes inferior reranking. When the candidate list improves, false positives can become harder to recognize as they tend to share confounding characteristics with the true positives. A discriminative reranker should be able to handle the top portion of retriever results and avoid relying on those confounding features.
In this paper, we introduce Localized Contrastive Estimation (LCE) learning. We localize negative sample distribution by sampling from the target retriever top results. Meanwhile, we use a contrastive form loss which penalizes signals generated from confounding characteristics, preventing the reranker from collapsing.
Experiments on the MSMARCO document ranking dataset show that LCE can better exploit the LMs capability. With the same BERT model, LCE achieves significantly higher accuracy without incurring training or inference overhead.

Background
Separation of retrieval into stages was introduced naturally due to efficiencyeffectiveness trade-off among different ranking models: fast but less accurate model (e.g. BM25) retrieves from the entire corpus while slower but more accurate ones (e.g. BERT) refines ranking in the top candidate list.
Heuristic retrievers like BM25 use matching signals exclusively from exact match and therefore can use inverted list data structure for low latency full corpus retrieval. They are limited by document statistics for scoring. As a fix, deep language models can be leveraged to re-estimate term weights in search index [6,5]. An alternative is adding probable query terms to document [12].
Pre-trained deep LMs [14,7] have demonstrated strong supervised transfer performance on reranking tasks. Popular recent works [4,10] fine-tune BERT [7] with binary classification objective and show it significantly outperforms earlier models. In this paper, we however question if this simple paradigm is sufficient to realize BERT's full potential, especially for high performance deep retrievers that generate candidates consisting of harder negatives.
Alternatives to binary classification objective are the contrastive learning objectives that directly take negatives into account [8]. The popular NCE loss computes scores of a positive instance and several negatives instances, normalize them into probabilities and train the model to give higher probability to the positive instance [16]. The incorporation of negatives in loss prevents the model from collapsing. While contrastive loss has been widely studied in representation learning [16,2], there are few prior works adopting it to train deep LM rerankers.

Methodologies
Preliminaries We aim to train a BERT reranker to score a query document pair, where cls extracts BERT's [CLS] vector and v p is a projection vector. We refer to the training technique popularly adopted [4,10] as the Vanilla method. It samples query document pairs independently and compute on each individual querydocument pair using binary cross entropy (BCE) based on query q document d and corresponding label (+/-), Vanilla method treats reranker training as a general binary classification problem. However, reranker is unique in nature; it deals with the very top portion of retriever results, each of which may contain many confounding signatures. The reranker is expected to, -Exile at handling top portion of retriever results.
-Avoid collapsing onto matching with confounding features.
To this end, in this section, we introduce Localized Contrastive Estimation (LCE) loss. The contrastive loss prevents collapsing and localized negative samples focus the reranker on top retriever results.
Localized Negatives from Target Retriever Given a target initial stage retriever and a set of training queries, we use the retriever to retrieve from the entire corpus, generating a set of document rankings for the queries. For each query q then sample from the set R m q of top ranked m documents, n nonrelevant documents as negatives examples. All sampled documents together form the negative training set. As will be shown in section 6, re-building training set based on the specific target retriever is critical to ensure robust training.
Contrastive Loss After aggregating all negatives sampled from target retriever, we form for each query q a group G q with a single relevant positive d + q and sampled non-relevant negative documents from R m q . We treat the BERT scoring function as a deep distance function, dist(q, d) = score(q, d) = v p cls(BERT(concat(q, d))) with which we define the contrastive loss for one query q as, Importantly, here loss and gradient condition not only on the relevant pair but also the retrieved negatives. This effectively helps prevent collapsing onto simple confounding matchings.

LCE Batch Update
Putting it all together, we can define the Localized Contrastive Estimation (LCE) loss on a training batch of a set of query Q as, Compared to a standard noise contrastive estimation (NCE) loss, LCE uses the target retriever to localize negative samples and focus learning on top portion instead of randomly sampled noisy negatives.

Experiment Methodologies
Dataset and Tasks We use the MSMARCO [1] document ranking dataset. The dataset contains 3 million documents. A document consists of 3 fields (title, URL, and body) with around 900 words. Models are trained on the train set of 0.37M training pairs. As recommended by MSMARCO organizers, we use the dev set for analysis.
Initial Stage Retriever We experimented with four initial retrievers: Indri, un-tuned BM25, tuned BM25 (denoted as BM25*), and HDCT [5]. The Indri search results come from MSMARCO organizers 2 . We build BM25 indices with the Anserini toolkit [17], from which we produce two sets of search results with the toolkit's default BM25 parameters and a set of tuned parameters suggested by the toolkit authors 3 . HDCT is the SOTA method for augmenting document search indices with term weights re-estimated with BERT; we use the rankings provided by the authors 4 . We input top 100 candidate lists to rerankers.
Implementation Following [4]'s BERT-FirstP setup, we input the concatenated document title, url and body's first 512 queries to the rerankers. Our rerankers are built and trained in mixed precision with PyTorch [13] and based on Huggingface's BERT implementation [15]. We sample negatives from the target retriever's top ranked m = 100 documents similar to reranking depth. We train on 4 RTX 2080 ti GPUs, each with a batch of 8 documents. We train for 2 epochs, with a 1e-5 learning rate and a warmup portion 0.1.

Document Ranking Performance
In table 1, we summarize ranking performance on MSMARCO document ranking Dev and Eval (leaderboard) queries. Here, both vanilla and LCE use negatives from target retriever. On the dev set, we test rerankers trained with vanilla and LCE loss on each type of the first-stage retriever. We see LCE significantly improves performance with all retrievers. Meanwhile, we see that gain using LCE enlarges as the retriever grows stronger, suggesting it can capture more complicated matching in the improved candidate list, while not being confused by the harder negatives.
The leader board results confirmed the effectiveness of LCE. HDCT+LCE pipeline outperformed the vanilla basline by a large margin. Following other recent leaderboard submissions, we further incorporate model ensemble. Our ensemble entry uses an LCE trained ensemble of BERT, RoBERTa [9] and ELEC-TRA [3] to rerank HDCT top 100. This submission got first place, achieving the state-of-the-art performance 7 .

Analysis
In this section, we first analyze the effect of number of sampled documents per query in LCE, then the influence of the negative sample localization. Effect of LCE Sample Size In Figure 1, we study the effect of varying the number of sampled documents per query in the LCE loss. We observe a big improvement from size 2 (1 positive, 1 negative) that compute loss scale with a single negative, to size 4 where loss weights are computed with 3 negatives. Further increase in sample size can generate some additional improvements.
Influence of Negative Localization LCE samples negatives from top ranked documents retrieved by target retriever. Here we quantitatively evaluate its importance. Denote retriever used in training for negative sampling train retriever and in testing for candidate generation test retriever. We use all rerankers from section 5 to rank candidate lists generated by all retrievers and plot results in a heat map Figure 2. We plot rerankers using different train retrievers on the horizontal axis and test retrievers on the vertical axis. Each 4 × 4 sub-grid corresponds to a training strategy, and sub-grid diagonals correspond to section 5 results. We observe localization benefits both LCE and vanilla methods. rerankers trained with LCE loss also perform better with localization. Interestingly, we do find that the LCE loss can bring some degrees of adaptability to the reranker, making it robust when the test retriever is different from the train retriever.

Conclusion
Recent research shows promising results on using deep LMs to improve initial retrievers. However, we discovered that previous BERT rerankers could not fully exploit the improved initial rankings. We propose Localized Contrastive Estimation (LCE) learning, to localize training negatives with target retriever, and to use a contrastive loss to penalize matching with confounding characteristics. Experimental results demonstrate that reranker trained with LCE significantly outperforms its vanilla method trained counterpart using the same LM. Our analysis shows that localizing negatives and having an expressive loss with multiple contrastive negatives are both critical for the success of LCE.
The positive results show that, instead of adopting more advanced LM, it is also possible to improve the performance of existing deep LMs with better learning methods. Meanwhile, before this work, there are few existing work studying the interaction between different deep retrievers and reranker in pipelined retrieval systems. We believe this paper will encourage the community to conduct more systematic research on pipelined IR systems.


Image-based scene recognition for robot navigation considering traversable plants and its manual annotation-free training

This paper describes a method of estimating the traversability of plant parts covering a path and navigating through them for mobile robots operating in plant-rich environments. Conventional mobile robots rely on scene recognition methods that consider only the geometric information of the environment. Those methods, therefore, cannot recognize paths as traversable when they are covered by flexible plants. In this paper, we present a novel framework of image-based scene recognition to realize navigation in such plant-rich environments. Our recognition model exploits a semantic segmentation branch for general object classification and a traversability estimation branch for estimating pixel-wise traversability. The semantic segmentation branch is trained using an unsupervised domain adaptation method and the traversability estimation branch is trained with label images generated from the robot's traversal experience during the data acquisition phase, coined traversability masks. The training procedure of the entire model is, therefore, free from manual annotation. In our experiment, we show that the proposed recognition framework is capable of distinguishing traversable plants more accurately than a conventional semantic segmentation with traversable plant and non-traversable plant classes, and an existing image-based traversability estimation method. We also conducted a real-world experiment and confirmed that the robot with the proposed recognition method successfully navigated in plant-rich environments.

I. INTRODUCTION
The mobile robot technologies have come nearly to the level of practical realization and commercialization in some situations. For example, self-driving cars are already being tested in public areas [1,2]. Another example is the service robots operating in public places such as airports and stores [3,4]. Those applications mainly target structured environments such as urban areas and the inside of buildings. In plantrich environments such as agricultural fields and forest paths, however, it is more difficult for robots to autonomously navigate than in structured environments because the paths are possibly covered by plant parts such as branches and leaves, which, though, can be driven through by the robots. Most of conventional mobile robots only consider the presence of objects and do not consider such traversable objects.
One of such environments is greenhouses and agricultural fields. Traditionally, in such environments, plants are extracted from sensor readings and then two plant rows that form a path are detected via simple line fitting algorithms [5,6,7,8,9]. Those methods assume that paths are straight and open, and not covered by any plant parts. A recent work [10] applies an end-to-end policy learning to learn a function that directly maps an image to an appropriate control. Although their method succeeded to navigate in outdoor scenes with VOLUME 1, 2021

Semantic Segmentation Module Traversability Estimation Module
Trained with Pseudo-labels → Generated using public datasets  tall grass, it requires "trial and error" experiences achieved via a robot's random walk in the real environments, which are difficult to acquire in environments like greenhouses and other environments with objects that the robot must not collide with. Imitation learning (IL) is another possible approach for end-to-end policy learning that takes a human's demonstration of control for supervision. However, IL requires abundant amount of high-quality demonstrations given by experts, which are not easy to collect. Moreover, even when using learned control policies, explicit scene recognition is still important for safe deployment of navigation systems in such environments. Therefore, we aim at an image-based scene recognition considering traversable plants for robot navigation in plant-rich environments such as greenhouses.
In this paper, we present a novel framework of scene recognition and robot navigation that takes into account traversable plants covering the paths. To recognize objects as traversable, it is not sufficient to just recognize plants because plants have both traversable parts such as branches and leaves, and non-traversable parts such as stems. Datasets with such fine-grained classes are not publicly available to the best of our knowledge. To provide information of traversability of objects, which cannot be acquired from existing datasets, we use label data indicating the regions with objects that the robot has traversed during data acquisition phases, coined traversability masks. Since the traversability masks are generated based on a limited number of the robot's traversal experiences, all traversable regions are not labeled but some of them are left unlabeled (See Fig. 2). Directly using such masks as labels of traversable plants will confuse the training. To take full advantage of the incomplete information of the traversability masks, we present a deep neural network (DNN) architecture and its training method inspired by PU (positive and unlabeled) learning [11], where a model is trained with labeled positive examples and unlabeled examples including both positive and negative ones.
The overview of the proposed method is shown in Fig.  1. The proposed DNN architecture consists of Semantic Segmentation Module (SSM) for pixel-wise general object classification and Traversability Estimation Module (TEM) for estimating traversability which indicates how likely that each pixel can be traversed by robots. SSM is trained without a manually labeled greenhouse dataset by means of an unsupervised domain adaptation (UDA) method proposed by the authors [12]. TEM is then trained using the features from SSM as inputs and the traversability masks. For the training of TEM, we introduce a training method inspired by PU learning [11] in order to take full advantage of the information given by the traversability masks. The overall network is therefore trained without manual annotation of the training images.
The prediction results (the object class and the traversability of each pixel) are then projected to the 3D space to build a semantic 3D voxel map used in navigation. For each data frame, the prediction results (observations) are temporally fused with the prior probabilities of the object classes and the traversability in each voxel using Bayes' update. By treating as free spaces voxels where the probability of plant is the highest and the traversability is greater than a threshold, the robot is able to navigate through traversable plant parts covering the paths.
The contributions of this paper are as follows: 1) A novel framework of scene recognition considering both general object classes and the traversability of objects for robot navigation in plant-rich environments. 2) A manual annotation-free training method for the scene recognition model. 3) A PU learning-based training for traversability estimation which requires only positive labels given to a part of traversable regions in images. 4) Applying the scene recognition model to the navigation tasks in a real greenhouse with plants partially covering the paths.

II. RELATED WORK A. TRAVERSABILITY ESTIMATION 1) Geometry-based methods
Traverasability estimation is a crucial task in robot navigation. Traditionally, rigid obstacles and negative obstacles such as ditches and holes are detected using geometric information from range sensors such as laser range finders (LRFs), stereo cameras, and RGB-D sensors. In unstructured scenes, many studies have focused on estimating the traversability of terrains by analyzing the steepness, roughness and so on [13,14,15,16]. Bogoslavsky et al. [13] estimated the traversability of indoor environments using Kinect sensor by analyzing the normals. Zhu et al. [14] exploited disparity information from a stereo camera to estimate traversable regions consisting of a flat plane. Martínez et al. [15] used static obstacle detection and terrain analysis using 2D LRF in their reactive navigation scheme. The near-to-far approach is also a popular approach, where the terrain traversability is estimated within the measurement range of a range sensor and the estimated traversability is then propagated to farther areas based on the similarity of image features [17,18,19]. The researches mentioned above are for estimating the traversability of the terrain, i.e., whether the robot can drive over the terrain. Unlike those methods, we deal with the estimation of the traversability of objects such as plant parts hanging over and covering the paths, i.e., whether the robot is allowed to touch and push them while navigating.

2) Semantics-based methods
Other methods exploit object semantics estimated from RGB images [20,21,22,23,24,25]. The rich information of RGB images such as colors and texture allows for estimating object semantics such as object class and traversability from the appearance. Zhou et al. [20] trained a terrain classifier on visual features with class labels automatically given using structural information from a 3D LiDAR. Matsuzaki et al. [21] used examples of traversable path lines given by human annotators to train a probabilistic model to classify the image regions into either of the object classes: building, road, or grass. In the last decade, DNNs played a crucial role to provide high recognition accuracy and generalization ability. Maturana et al. [22] generated a 2.5D semantic map using DNN-based semantic segmentation for path planning of autonomous off-road vehicles. Gao et al. [23] proposed a fine-grained traversability estimation to deal with different level of traversability of the terrain. Guan et al. [24] proposed a hybrid method of DNN-based semantic segmentation and analysis of geometric information for autonomous navigation of excavators.
None of the aforementioned methods, however, addressed traversable objects such as plants covering the paths, which are the main target of our method. Although the authors' previous work [26] constructed a semantic 3D map inside a greenhouse, traversable and non-traversable plants are not distinguished and robot navigation was not realized. In addition, those DNN-based methods to estimate object semantics require an abundant amount of manual annotation by humans, while our method provides manual annotation-free training.

3) Self-training based on experience
Using information of robot's traversal is a preferred approach for identifying traversable regions in recent studies. Barnes et al. [27] used trajectories of an autonomous vehicle for automatically labeling images for semantic segmentation. Similarly, Wellhausen et al. [28] proposed to use footprints of a legged robot to label images and showed that sufficient performance can be achieved even with such sparse labels. Lee et al., [29] trained a multi-layer perceptron (MLP) on geometric features in a self-training manner with data labeling using both a robot's trajectory and structural information from an elevation map. Those methods are, however, not applicable to traversable objects hanging over threedimensionally. Some studies consider the traversability of objects as an affordance depending on both the environment and the robots. Kim et al. [30] trained a classifier with simple hand-crafted features of image patches by exploiting a robot's experience of successful and unsuccessful traversals acquired by its interaction with objects. In another work [31], they proposed a superpixel-based image classification to improve the accuracy of region segmentation. The studies by Kim et al. share the most similar challenge with our work in terms of considering the traversability of plants.
Recently, methods based on Reinforcement Learning (RL) and Imitation Learning (IL) have also been presented. Kahn et al. [10] applied in outdoor robot navigation such an end-toend policy learning based on robot's experience of trial and error gathered through its random walk motion in real environments. Despite their success, we claim that such a training method is not suitable for navigation in greenhouses or other plant-rich environments because there may be objects that the robot must not hit such as crop rows and equipments, or steep slopes beside the path. Random motions in such environments may lead to a fatal failure. IL methods instead exploit high-quality demonstrations by human experts. Such demonstrations are, however, not always available. Even when a kind of policy learning can be used, explicit scene recognition is still necessary for ensuring the safety. In this sense, our proposed method can be rather complementary to those methods. VOLUME 1, 2021

B. AGRICULTURAL MOBILE ROBOTS
Agricultural mobile robots are one of the practical and actively studied applications of mobile robots in plant-rich environments. Many of currently used agricultural robots move on rails or heating pipes equipped on the paths [32,33]. Robotic systems with such locomotion is able to move stably, while movable areas for operation are limited. In addition, such systems require installation of the rails when not installed. More flexible robot operation is beneficial for extending the field of tasks that the robots can automate.
In outdoor, precise localization by GNSS is adopted in various studies [34,35]. In indoor fields such as greenhouses, however, such a method is not available due to noise imposed by surrounding obstacles. Therefore, a popular approach is recognizing a traversable path using sensors equipped on the robot. The sensors used in path recognition include ultrasonic sensors [36,37], LIDARs [7,38,9] and monocular / stereo cameras [39,5,6,8].
Most of those methods detect two plant rows that form a path with a combination of plant detection based on sensor readings, and line fitting algorithms such as Hough transformation [5,8] and iterative methods like RANSAC [9,7]. Those methods assume that the lines of plants are visible. However, this is not the case in environments where some plant parts cover the path. In one of pioneering work by Mandow et al. [36], it has been reported that while they succeeded to autonomously operate a mobile robot using an ultrasonic sensor, some failure cases observed where the robot recognized plant parts overhanging the path as obstacles and stopped the navigation. To the best of our knowledge, the problem of recognizing traversable plant parts covering the paths has not been explicitly addressed.

C. SEMANTIC SEGMENTATION
Semantic segmentation is a task to assign an object class to each pixel of an image. Long et al. [40] first proposed to convert classification networks into fully convolutional network to produce a probability map for each pixel. Based on ResNet proposed by He et al. [41] as a backbone network, very deep networks have been shown to achieve high accuracy [42,43]. While it is proven to be effective to stack layers in depth to get a high accuracy, those models impose high computational cost. For real-time visual tasks, computationally efficient architectures have also been studied, targeting the applications such as robotics and autonomous driving systems [44,45,46]. Recent approaches further improved the accuracy while maintaining the efficiency by introducing more efficient convolution approaches such as factorized convolution [47], depth-wise separable convolution [48], efficient spatial pyramid [49,50] etc.
In our previous work, we have shown that a semantic segmentation network for greenhouses can be trained without requiring hand-labeled images, by utilizing multiple publicly available datasets of urban scenes and employing an unsupervised domain adaptation (UDA) method [12].

D. PU LEARNING
PU (positive and unlabeled) learning is a problem where part of the data is labeled as positive, and the rest is unlabeled and could be either positive or negative. Unlike the setting where positive and negative data are assumed, PU setting cannot be solved by just applying a simple classification of positive and unlabeled data since the unlabeled data include positive ones. Elkan and Noto [11] showed that it is possible to estimate the label probabilities by modeling the probability that given samples are labeled. As a practical application, Yang et al. [51] applied PU learning framework in object detection task where some of objects are not labeled.
In our work, we exploit binary-labeled images where pixels that the robot has traversed are labeled, and the rest is left unlabeled. We can treat the labeled pixels as "traversable" based on the robot's traversals, whereas it is not always true that the unlabeled pixels are "non-traversable", since there is a substantial amount of unlabeled pixels of traversable objects as later shown in III-A. This problem is exactly an example of PU learning. We therefore introduce a training method inspired by Elkan and Noto [11] to the training of TEM in III-B2. Fig. 1 shows the overview of the proposed framework. We employ two network modules for the scene recognition: Semantic Segmentation Module (SSM) for semantic segmentation, and Traversability Estimation Module (TEM) for traversability estimation. SSM is trained by an unsupervised domain adaptation method. For the training of TEM, we generate traversability masks based on the robot's traversal experience. The network is then trained with the generated traversability masks ( Fig. 1(a)). Predictions of the trained network are projected to the 3D space to build semantic local 3D map that indicates the object classes and traversability of the regions around the robot and the navigation is operated based on the map ( Fig. 1(b)).

A. TRAVERSABILITY MASK
The traversability masks are label data that indicate the image regions traversed by the robot or a human during the data acquisition phase. Using the traversability masks, we automatically label the regions of images dominated by traversable plant parts to train a network so that it can distinguish traversable and non-traversable plants based on the robot's experience.
We first acquire RGB-D data by manually operating a robot or walking inside the target greenhouse with an RGB-D sensor moving through the plants. We then build a 3D voxel map using RTAB-Map [52] from the RGB-D data. After that, by utilizing the robot's path estimated in the mapping process, the voxels traversed by the robot or the human are labeled. Here, we approximate the shape of the robot or the human with a rectangular, and from each observation point of the 3D voxel map, voxels within the rectangular are labeled as traversed. Finally, traversability masks are generated by Examples of the traversability masks are shown in Fig. 2. One may notice that while much of traversable plant parts such as leaves and branches are labeled, there is also quite a lot of unlabeled regions of traversable plants. This is because the traversability masks are based on the limited number of traverses during the data acquisition phase. This nature of the traversability masks makes it inappropriate to simply treat them as positive and negative examples. By our training method inspired by PU learning [11] described in III-B2, the traversability masks are effectively exploited to distinguish traversable and non-traversable plants.

B. NETWORK ARCHITECTURE
In this section, we describe a network architecture for pixelwise semantic segmentation and traversability estimation. The overview of our network is shown in Fig. 1(a). The network consists of Semantic Segmentation Module (SSM) for pixel-wise object label assignment and Traversability Estimation Module (TEM) for estimating how likely that each pixel can be traversed by robots. SSM is first trained independently. TEM is then trained with intermediate features of SSM as inputs and traversability masks as training labels. The reason for not training them simultaneously is to avoid overfitting of the entire network to the traversability masks which include unlabeled traversable regions.
Our network is based on ESPNetv2 [50] and we make two modifications. First, an auxiliary segmentation branch is attached to the middle of the main network of ESPNetv2. We refer to the segmentation network including the auxiliary branch as SSM (Semantic Segmentation Module). The auxiliary branch is used to estimate pixel-wise uncertainty for the training of SSM proposed in [12]. Second, a branch for estimating traversability, i.e., TEM is attached. The interme-diate feature from the SSM is fed into a 3 × 3 convolution and the sigmoid function to produce pixel-wise probability predictions. For the complete network architecture, see Fig.  8 in the Appendix. The additional layers do not affect the computational efficiency and still allows for prediction at sufficient speed as explained in IV-B.
The advantages of our architecture are three-fold. Firstly, the architecture enables the PU learning-based training to model the traversability as probability values. Using the semantic segmentation task provides discriminative features for the PU learning. Secondly, by separating the tasks into semantic segmentation and traversability estimation, the branches for each task can be trained with the maximum possible number of data. Because the traversability masks are generated from a map, the number of available training data for TEM is less than that for SSM, which only requires RGB images. Thanks to the network structure, the SSM can fully exploit available training images for a better performance of semantic segmentation individually from the training of TEM. In fact, as shown later in IV-A, the SSM was trained with 6689 images while the TEM was trained with 889 images with traversability masks in our experiment. Thirdly, the two modules provide complementary information. Misclassifications of TEM can be refined by a prediction of SSM by utilizing prior knowledge that, e.g., artificial objects are not traversable. We evaluate this effect in IV-A4.

1) Semantic Segmentation Module
Semantic Segmentation Module (SSM) is responsible for pixel-wise general object classification. Here we use three general object classes: plants, artificial objects, and the ground. For training SSM, we introduce a training method in our previous work [12]. In the method, we employ multiple publicly available datasets with pixel-wise labels (the source datasets) and an unlabeled greenhouse dataset (the target dataset). First, segmentation models are trained with each source dataset. Images from the target dataset are then fed into the models and corresponding outputs are yielded. If a pixel is assigned with the same object label from all the source models, the label is treated as a pseudo-label of the pixel. For training of SSM with the pseudo-labels, we adopt a method of adaptively weighting a loss value on each pixel based on the uncertainty of the prediction using the main and auxiliary branches [53].

2) Traversability Estimation Module
Traversability Estimation Module (TEM) estimates the probabilities that the pixels are traversable. In the training of TEM with the traversability masks, we introduce the PU (positive and unlabeled) learning framework. The purpose of PU learning is to model a probability function p(y = 1|x), where x is an input data and y ∈ {0, 1} is a binary label of the data x, in a situation where a part of the positive data (y = 1) is labeled as positive, and the rest is unlabeled and can be either positive or negative. Elkan and Noto [11] state that under the "selected completely at random" assumption, meaning that the labels VOLUME 1, 2021 are given completely at random, PU learning can be solved by modeling a classifier g(x) such that g(x) = p(s = 1|x), where s denotes a random variable indicating that data x is labeled if s = 1. Here, note that y = 1 if s = 1 but not vice versa. The equation can then be transformed as follows, given the "selected completely at random" assumption, i.e., p(s = 1|y = 1, x) = p(s = 1|y = 1): Therefore, the traversable probability of a pixel p(y = 1|x) can be calculated as follows: where c = p(s = 1|y = 1) which denotes the conditional probability that a positive data is labeled. c is approximated by feeding the training data to the estimated g(x) and average the probabilities of the labeled features, i.e., where P denotes a set of all labeled pixels in the training dataset and n is the number of elements in P .
In TEM, we first model the label probability g(x) = p(s = 1|x) by a convolution and a sigmoid activation. The output is then divided by c calculated by eq. (3) to produce p(y = 1|x). The feature map from the SSM is fed in TEM. The size of the feature map is (B, C, H, W ) where B denotes the batch size, C denotes the channel size, H and W denote the height and the width, respectively. Here we consider a vector of size C as an input x for the corresponding pixel. 3 × 3 convolution is applied to the features, followed by a sigmoid function to scale the output in the range of [0, 1]. We use 3×3 convolution instead of 1 × 1 convolution to take information of adjacent pixels into consideration in the estimation.
Note that the "selected completely at random" assumption does not strictly hold in our task. In practice, however, we confirm that this formulation is effective to estimate pixelwise traversability.

C. 3D SEMANTIC VOXEL MAP
Using the results of the recognition explained in section III-B, we build 3D semantic voxel map around the robot. At first, an RGB image acquired from the RGB-D sensor is passed to the segmentation network. The predictions from SSM and TEM are then mapped into the 3D space using the corresponding depth image. The 3D space around the robot is divided into voxels with a side of 0.1 [m], and the predictions mapped into the 3D space are assigned to a corresponding voxel that they fall into. For the object class information, a histogram of the object classes is constructed in each voxel and the class label with highest frequency within the voxel is used as an observation. For the traversability information, the traversability values of the points within a voxel are averaged and treated as an observation.
In each time step, those observations are temporally fused by Bayesian update. The observation of the object class z o t at time t is fused by the following equation: where l t denotes an object label of the voxel at time t, η is a normalization term and P (l t−1 ) is the prior of label l t−1 at time t−1. P (z o t |l) is the likelihood of label l with observation z o t . The likelihood is calculated using greenhouse images not used in the training of SSM as follows. Firstly, pseudolabels are generated for each image. The same images are then fed in SSM and outputs of pixel-wise class label are produced. For each object class l, a histogram of predicted object classes over all pixels of the pseudo-labels with the label l. A conditional probability P (z o |l) is calculated by normalizing the histogram and used as a likelihood function. The object class with the highest probability is assigned to the voxel as its label.
Similarly, the observation of traversability is fused by the following equation: where τ t ∈ {0, 1} denotes an event that the voxel is traversable (1) or not traversable (0) at time t. The likelihood P (z τ t |τ ) is calculated using the training images with traversability masks as follows. The input images are fed in TEM and outputs of traversability values are yielded. For each label τ ∈ {0, 1} in the traversability masks, a histogram of estimated traversability values is calculated over all pixels with the traversability label τ . The histograms are then normalized to calculate a conditional probability P (z τ t |τ ) and it is used as a likelihood.
After the integration of observed points to the semantic voxel map, obstacle point cloud is generated by treating the voxels whose object class is "plant" and the traversability is higher than the threshold as free spaces and the others as obstacles. The location of a point for a voxel is the centroid of the points that have been accumulated in the voxel over the frames. By feeding the obstacle point cloud to a conventional navigation module, the navigation in plant-rich environments is realized. For mitigating the degradation of the processing speed as the number of voxels increases, the voxels where no point has been observed for a certain number of consecutive frames are removed from the map. In the experiment below, the number of frames is set to 10.

A. EVALUATION OF TEM 1) Used datasets
For the training of SSM by the UDA method, we used three source datasets: CamVid [54], Cityscapes [55], and Freiburg Forest [56], for generating pseudo-labels. As target data, we used 6684 unlabeled greenhouse images from Greenhouse A dataset, taken in a greenhouse growing tomatoes. For the training of TEM, we used 899 pairs of an RGB image and the corresponding traversability mask from Greenhouse B dataset, taken in the same greenhouse as Greenhouse A on a different date. The data were collected through a human operator's control of the robot for about 20 minutes in total. During the data collection, the robot traversed each of the paths with plant rows on both sides only once. The labels are therefore imbalanced as can be seen in Fig. 2. Approximately 40 to 50 % of the traversable plant regions were actually labeled. Greenhouse A and B have a different appearance due to a different level of growth of the plants. For the evaluation, we manually gave true labels of traversable regions on 33 images from Greenhouse A. Overview of the greenhouse datasets is shown in Table 1.

2) Training conditions and hyperparameters
The DNN model is implemented with PyTorch [57] and all of the training is performed on one NVIDIA Quadro RTX 8000 with 48GB of memory. In the training of SSM, the training epoch is set to 200 with a fixed learning rate of 5 × 10 −5 , and the batch size of 64. TEM is then trained with weights of SSM fixed. The number of epoch is set to 200 and the batch size is 64. We use cyclical learning rate scheduling [58]. The initial learning rate is 5 × 10 −5 and linearly increases by a factor of 10 in 10 epochs and then decreases to the original value in 20 epochs.

3) Baselines for traversability estimation
As baseline methods of image-based traversability estimation, we use the following methods. Semantic segmentation We train a semantic segmentation model with "traversable plant" class as well as the classes used in the training of SSM. A model identical to SSM is used as a semantic segmentation model. For training, we used the same dataset as the one used in the training of TEM (899 RGB images with traversability masks). Pseudo-labels are generated for each image with the three classes. Labels of the pixels with the "plant" class are then changed to "traversable plant" if the value of the corresponding traversability mask is 1 (traversable). As a result, the object classes used in the training are as follows: traversable plant, other plant, artificial object, and ground. We initialize the network via pre-training with Greenhouse A dataset. A constant learning rate of 1 × 10 −4 is used and an epoch size is 50.   of training a classifier based on a robot's experience of successes/failures of traversals, which shares the research target with ours. To evaluate the baseline method, we conducted 10-fold cross validation using the 33 test images with ground truth labels. We use those data instead of the training images with traversability masks for two reasons: 1. the baseline method assumes complete labels of traversable/nontraversable while the traversability masks are not complete and include unlabeled traversable regions. 2. the baseline method is for incremental online learning with several image frames instead of hundreds of training data. In the original work [31], ten consecutive frames of images are used for training.

4) Results
We evaluate the binary traversability images generated from the predicted traversability with different thresholds. Here we compare two types of binary images: raw and refined.
The raw binary images are generated by just binarizing the predicted traversability with a threshold. The refined binary images are generated by additionally setting all the pixels predicted as an artificial object or the ground to nontraversable. This refinement provides the filtering effect of false positive traversability predictions even in the case of high predicted traversability, which is equivalent to the calculation in the voxel described in III-C where the final decision of traversability is given considering both the object class and the predicted traversability. Fig. 3 shows a Precision-Recall curve as well as an IoU-Recall curve to analyze the performance of TEM. The best IoU was achieved when the threshold was 0.75. Therefore, we use this traversability  Prediction results of TEM. From the top: Camera images, Ground truth, predicted binary images with threshold 0.75, refined binary images, binary images generated from the ordinary semantic segmentation method ("traversable plant" pixels are shown in white), and binary images generated from the probability maps predicted by the baseline method [31]. Note that the traversable regions are not identical to the plant regions, but plant parts such as stems are not traversable. TEM is capable of distinguishing the traversable plants from the non-traversable ones (highlighted in yellow ellipses). The refinement by predicted object classes mainly worked as noise suppression (highlighted in red rectangles).
threshold in the rest of the paper. Table 2 shows the accuracy, precision, and recall of the model on the test set with the threshold of 0.75. From Fig. 3 and Table 2, we can see high recall and relatively low precision. The refined predictions resulted in better precision and IoU. This result indicates that fusing the predictions of object classes and traversability decreased the false positive rate and thus is suitable for safe robot operation compared to only predicting the traversability. Fig. 4 is the visualization of the prediction by TEM. Much of the traversable regions are classified as positive. In addition, some non-traversable plants are predicted correctly. This result shows that our model can learn the differences between traversable and non-traversable plants. In terms of the refined results, we cannot qualitatively see obvious improvements, but we confirmed that small prediction noise was suppressed. Note that these segmentation results are yielded from automatically generated traversability masks, which are incomplete and noisy. Even from such data, our PU learningbased training framework effectively learns the features of traversable objects, and provides reasonable estimation. Comparison to the semantic segmentation TEM provided better performance than the segmentation-only method in all the metrics. Especially, the IoU and the recall were significantly low in the ordinary semantic segmentation. In Fig.  4, we can see that the semantic segmentation method could not capture the traversable plant regions well. We suppose it is because the labeling of "traversable plant" regions does not cover all the traversable plant regions as mentioned in III-A and results in inconsistent labels with traversable plant regions labeled as non-traversable plant. Such labels confuse the training. Therefore, the segmentation-only method does not get along with the sparse traversable masks. On the other hand, our proposed method enables more accurate and effective training of the model to estimate traversability of image regions.
Comparison to Kim et al. We report the average of the metrics over the 10 different partitions for the cross-validation in Table 2. For each data partition, the threshold of the probability is determined so that it yields the best mean IoU for the test images. As a result, our proposed method outperformed the baseline in all the metrics even though the baseline model is trained with ground truth labels while our method is trained with noisy and incomplete traversability masks. As can be seen in Fig. 4, although the plant regions are mostly classified correctly, other regions of artificial objects and the ground are also classified as traversable. Moreover, the traversable leaves and branches and non-traversable stem parts are not distinguished and almost equally classified as traversable.
Our method is advantageous over the baseline by Kim et al. in three aspects. First, our PU learning-based method enables the training without negative labels that are difficult to acquire. In [30], negative labels are collected using the information of the robot actually bumping into an obstacle, which is not appropriate in environments such as greenhouses. Our learning framework allows for safer and more practical way of training the scene recognition model by utilizing only positive labels acquired via the control by a human operator. Second, our model provides semantics about object classes to complement the estimation of traversability. As can be seen in Fig. 4, there are a lot of regions wrongly classified as traversable in the prediction of the baseline method. There is no way to correct the misclassifications in the baseline, while our method can amend them using the prediction by SSM as mentioned above. Third, our method can utilize more discriminative features thanks to the DNN model trained in the task of semantic segmentation, which led to the better performance of the proposed method over the baseline even by the raw predictions of TEM.

B. NAVIGATION IN A GREENHOUSE
Finally, we conducted a real-world navigation experiment in a greenhouse. All the software is processed on a laptop with an Intel Core i7-6700HQ, an NVIDIA GeForce 960M, and 32 GB RAM. Revast Mercury is used as a robot platform (See Fig. 5(a)). As sensors, the robot is equipped with a Kinect v2, an RGB-D sensor, and Hokuyo UTM-30LX, a laser range finder.
An image of a path in the greenhouse is shown in Fig.  5(b). The length of the paths is approximately 7.5 [m]. The software for the DNN prediction is implemented on Robot Operating System (ROS) [59] with C++ and Libtorch, a C++ library for PyTorch. The frame rate of the prediction of the network is approximately 8 [fps] and the 3D mapping runs at approximately 5 [fps]. The frame rate of the original ES-PNetv2 is also approximately 8 [fps] on the same computer and thus our modification to the network does not affect the computational efficiency.
As a baseline for navigation, we also test a system that considers all the voxels as obstacles. It corresponds to using only geometric information, which is a major approach in most of the current mobile robots.

1) Obstacle detection in a simple forward motion
As a primary experiment, we conducted an experiment of obstacle detection using the proposed 3D semantic map during the robot's act of moving forward. We assume that the path is straight and constant control signals of linear velocity are given to the robot while the robot recognizes the traversability of objects in front. The linear velocity is set to 0.1 [m/s]. An obstacle point cloud is generated from the  We carried out 5 trials on the same path. The baseline navigation method consistently failed the task due to the part of plants grown out to the path. In contrast, our system was able to recognize the traversable plants and navigate through the path between the plant rows in all the trials. This result shows the ability of our scene recognition method to recognize the traversable plants covering the path. There were, however, some cases where the robot stops for a while and then restarts even though there was no obstacle. This was due to misclassification of voxels in front of the robots as obstacles. The misclassification of voxels stems either from the error in the image-based semantic segmentation or from the registration error between the predicted segmentation images and the corresponding depth images, which RGB-D sensors inevitably have. Errors temporarily occurring on a few pixels are suppressed by the voxelization and the Bayes' update. When the error consistently occurs on a large area, however, the corresponding voxel can be misclassified. Fig. 6 is the visualization of the semantic 3D maps generated during the experiment at different times in chronological order. There were a lot of plant parts partially covering the path. Our proposed method was able to recognize the traversable plants and to traverse them. Fig. 6(b) shows the case where the system wrongly recognized the regions in a far front of the robot as obstacles shown in the red voxels. The robot stopped moving in response to the obstacles. After observing a few data frames while stopping, the obstacle voxels were removed from the map and the robot resumed the navigation. We suppose the cause of this phenomenon as follows. When the misclassified regions are far from the robot, the segmentation results and the depth readings can be more noisy and it results in the misclassification of the voxels or creating voxels on wrong locations. In the case  in Fig. 6(b), it is more likely that the voxels were spawned on wrong locations due to the depth noise. As the robot approaches the voxels, both the segmentation and the depth values become more accurate and the voxels are updated with those observations. As a result, in the case in Fig. 6(c), the misclassified voxels were deleted from the voxel map because points in those voxels were not observed for the predefined number of frames (10 frames) when the robot was stopping near them.

2) Integration with move_base
We also conducted an experiment of applying our 3D semantic voxel map to navigation using move_base [60], a de facto standard navigation software in ROS [59].
The obstacle point cloud is fed in move_base as a sensor reading. In addition to the obstacle point cloud, a 2D laser range finder (LRF) is used to complement the blind spot of the RGB-D sensor and stabilize the navigation. Because of this purpose, the obstacle range of the LRF is limited to 1.0 [m]. The configuration of the LRF is the same in the baseline method. Sub-goals are given manually via Rviz, an interactive GUI tool for visualization.
Here, we define criteria for judging the success or failure of the trials. In all the trials, we observed that the accuracy of the SSM consistently degrades about 2.5 [m] before the end of the paths and the robot fails to detect the traversable path (We discuss this problem in IV-C). We therefore consider a trial in which the robot reached about 2.5 [m] before the end of the path as "traversed". When the robot stops due to misclassification, we allow a human's intervention by manually restarting move_base to clear the costmap at most once. Trials where the robot gets stuck even after the human intervention are marked as "stuck".
The baseline navigation method consistently failed to find a path on three paths before entering them. An example of a costmap generated with the baseline method is shown in  Fig. 7(b). We carried out 12 trials of navigation on the three paths (4 each) with the proposed method. The result is shown in Table 3. The robot with the proposed method succeeded to traverse through the path partially covered by plants in 8 trials, 5 of which involved a human's intervention. The costmaps generated with the proposed method are compared in Fig. 7(c). From these results, we can see that our proposed method enables the robot to recognize the traversable plant parts, which would otherwise be recognized as obstacles, and traverse through them. However, misclassifications affected the robot's motion as can be seen from the number of required interventions and failures.

1) Cause of the failures of navigation
We showed that our method enabled navigation through the paths covered by traversable plant parts in the greenhouse, while the baseline method constantly failed. In all the trials, however, the robot with our proposed method consistently failed to navigate about 2.5 [m] before the end of the paths. In addition, in some of the successful trials, it required a human's intervention due to misclassifications. On the other hand, in the experiment in IV-B1, all the 5 trials were successful without any interventions.
We suppose that the difference of the degrees of the success rates stems from the difference of the density of leaves. In such a case, majority of the regions are dominated by a single class of "plant" and a few misclassifications are filtered out in the probability calculation in each voxel. In the experiment in IV-B2, however, leaves were sparse and artificial objects behind the plant rows were also visible (a) Input image (b) Baseline for navigation (c) Ours through the leaves. This was especially obvious near the end of the path. Since the SSM is trained with pseudo-labels, which are generated by merging the outputs from multiple pre-trained models, the model lacks pixel-level accuracy. This is the current limitation of our method. For more reliable navigation, the accuracy of the recognition model needs to be improved. In addition, to deal with the prediction noise in DNN models, a better filtering and/or refinement methods should be implemented.

2) Adaptability of the proposed system
The success of the training of our scene recognition model depends on the UDA of semantic segmentation, since TEM takes the intermediate features of the semantic segmentation network (SSM) for estimation. In terms of the segmentation tasks in greenhouses, we confirmed in the previous work [12] that our UDA method is applicable to multiple greenhouses and multiple seasons. The proposed method will thus also be applicable to various greenhouse environments. When applying the proposed system in a new environment, in general, the segmentation model will need to be newly trained to adapt to the environment. Once the segmentation model is trained, TEM can be trained in the same manner as described in this paper. Note that our proposed model and the training method provide advantages of the training with a little images and incomplete positive labels, which lead to a minimum burden in deployment.

3) Influence of the sensor setting
In this work, we employed a Kinect v2 for simplicity of implementation of the sensor system to acquire calibrated RGB and depth images. Although it is sensitive to disturbance from strong light, we did not observe any severe effect of the sunlight on the depth readings. in our experiments. We suppose it is because the sunlight was cut by the translucent roof and walls of the greenhouse and the leaves, and thus the sensor was able to work in moderate lighting conditions. When we apply the system to outdoor environments, however, the choice of sensors will influence the accuracy and reliability of the system. Note that our proposed method is compatible with any sensors that can provide registered RGB images and depth readings and the sensors can be selected based on the lighting condition of the target environment. Applicable sensor settings include a stereo sensor, and wellcalibrated 3D LiDAR and a monocular camera.
In terms of sensor calibration, Kinect v2 was used out-ofthe-box in our experiments. In multi-sensor setting, it will indeed require a precise calibration, although the Bayes' update and the voxelization process can to some extent deal with a little noise due to the misalignment of the sensors.

V. CONCLUSIONS
In this paper, we described a method of estimating the traversability of plants covering a path and navigating through them in plant-rich environments for mobile robots. We proposed the following novel methods. 1. A scene recognition model that estimates general object classes and the traversability of the objects. 2. A manual annotation-free training of the model using an unsupervised domain adaptation method for the semantic segmentation module (SSM) and the information of the robot's traversals, named Traverasability masks for the traversability estimation module (TEM). 3. A PU learning-based training method to effectively train TEM with the traversability masks, which include unlabeled traversable regions. Our proposed method enables easy and safe deployment of mobile robots with the capability of recognizing traversable plants with a minimum burden. In the comparative analysis, we confirmed that the proposed method is able to estimate the traversability of image regions more accurately than a conventional semantic segmentation and an existing work of image-based traversability estimation. Moreover, we applied the scene recognition model to the navigation tasks in a real greenhouse with plants covering the paths and confirmed its ability to recognize traversable plants and navigate through them.
As future work, we are going to work on the improvement of the navigation system. In particular, the accuracy of the scene recognition and the filtering of misclassification should be improved. We are also looking to applying this work to a navigation task in unstructured environments such as forest VOLUME 1, 2021 paths. . Fig. 8(a) shows a complete structure of our network architecture. Our network is based on ESPNetv2 [50], a light-weight semantic segmentation network. It consists of modules such as Extremely Efficient Spatial Pyramid (EESP), Efficient Point-Wise Convolution (EffPWConv), Efficient Pyramid Pooling (EffPyrPool), which consist of several convolutions with different types of kernels, batch normalization, and activation layers. For detailed descriptions of those modules, we refer the readers to [50]. An auxiliary segmentation branch and a branch for estimating traversability, i.e., TEM are attached to the middle of the main network of ESPNetv2.

APPENDIX A DETAILS OF THE NETWORK STRUCTURE
The features for TEM are generated by concatenating the intermediate features from the EffPyrPool layer in each segmentation branch (See Fig. 8(b)). The intermediate feature from the SSM is fed into a 3 × 3 convolution and the sigmoid function to produce pixel-wise probability predictions.  The numbers beside each layer shows the spatial resolution relative to the input image and the channel size of the output of the layer, respectively. Based on ESPNetv2 [50], an auxiliary segmentation branch is added to estimate pixel-wise uncertainty for the training of SSM proposed in [12]. For estimating traversability, the intermediate features of the two segmentation branches are concatenated and fed into a 3 × 3 convolution followed by the sigmoid function. For the detail of each network layer, refer to [50].