extfieldsofstudy:
Engineering
Computer Science

text:
Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels

Recent studies on contrastive learning have achieved remarkable performance solely by leveraging few labels in the context of medical image segmentation. Existing methods mainly focus on instance discrimination and invariant mapping. However, they face three common pitfalls: (1) tailness: medical image data usually follows an implicit long-tail class distribution. Blindly leveraging all pixels in training hence can lead to the data imbalance issues, and cause deteriorated performance; (2) consistency: it remains unclear whether a segmentation model has learned meaningful and yet consistent anatomical features due to the intra-class variations between different anatomical features; and (3) diversity: the intra-slice correlations within the entire dataset have received significantly less attention. This motivates us to seek a principled approach for strategically making use of the dataset itself to discover similar yet distinct samples from different anatomical views. In this paper, we introduce a novel semi-supervised 2D medical image segmentation framework termed Mine yOur owN Anatomy (MONA), and make three contributions. First, prior work argues that every pixel equally matters to the model training; we observe empirically that this alone is unlikely to define meaningful anatomical features, mainly due to lacking the supervision signal. We show two simple solutions towards learning invariances - through the use of stronger data augmentations and nearest neighbors. Second, we construct a set of objectives that encourage the model to be capable of decomposing medical images into a collection of anatomical features in an unsupervised manner. Lastly, our extensive results on three benchmark datasets with different labeled settings validate the effectiveness of our proposed MONA which achieves new state-of-the-art under different labeled settings.

Introduction
With the advent of deep learning, medical image segmentation has drawn great attention and substantial research efforts in recent years. Traditional supervised training schemes coupled with large-scale annotated data can engender remarkable performance. However, training with massive high-quality annotated data is infeasible in clinical practice since a large amount of expert-annotated medical data often incurs considerable clinical expertise and time. Under such a setting, this poses the question of how models benefit from a large amount of unlabelled data during training. Recently emerged methods based on contrastive learning (CL) significantly reduce the training cost by learning strong visual representations in an unsupervised manner [12,18,19,35,36,38,40,41,62,66,91]. A popular way of formulating this idea is through imposing feature consistency to differently augmented views of the same imagewhich treats each view as an individual instance.
Despite great promise, the main technical challenges remain: (1) How far is CL from becoming a principled framework for medical image segmentation? (2) Is there any better way to implicitly learn some intrinsic properties from the original data (i.e., the inter-instance relationships and intrainstance invariance)? (3) What will happen if models can only access a few labels in training?
To address the above challenges, we outline three principles below: (1) tailness: existing approaches inevitably suffer from class collapse problems -wherein similar pairs from the same latent class are assumed to have the same representation [2,21,50]. This assumption, however, rarely holds for real-world clinical data. We observe that the long- tail distribution problem has received increasing attention in the computer vision community [23,44,46,97,113]. In contrast, there have been a few prior long-tail works for medical image segmentation. For example, as illustrated in Figure 1, most medical image images follow a Zipf long-tail distribution where various anatomical features share very different class frequencies, which can result in worse performance; (2) consistency: considering the scarcity of medical data in practice, augmentations are a widely adopted pre-text task to learn meaningful representations. Intuitively, the anatomical features should be semantically consistent across different transformations and deformations. Thus, it is important to assess whether the model is robust to diverse views of anatomy; (3) diversity: recent work [3,84,110] pointed out that going beyond simple augmentations to create more diverse views can learn more discriminative anatomical features. At the same time, this is particularly challenging to both introduce sufficient diversity and preserve the anatomy of the original data, especially in data-scarce clinical scenarios. To deploy into the wild, we need to quantify and address three research gaps from different anatomical views.
In this paper, we present Mine yOur owN Anatomy (MONA), a novel contrastive semi-supervised 2D medical segmentation framework, based on different anatomical views. The workflow of MONA is illustrated in Figure 2.
The key innovation in MONA is to seek diverse views (i.e., augmented/mined views) of different samples whose anatomical features are homogeneous within the same class type, while distinctive for different class types in imbalanced and unlabeled clinical scenarios. We make the following contributions. First, we consider the problem of tailness. An issue is that label classes within medical images typically exhibit a long-tail distribution. Another one, technically more challenging, is the fact that there is only a few labeled data and large quantities of unlabeled ones during training. Intuitively we would like to sample more pixel-level representations from tail classes. Thus, we go beyond the naïve setting of instance discrimination in CL [18,35,38] by decomposing images into diverse and yet consistent anatomical features, each belonging to different classes. In particular, we propose to use pseudo labeling and knowledge distillation to learn better pixel-level representations within multiple semantic classes in a training minibatch. Considering performing pixel-level CL with medi-cal images is impractical for both memory cost and training time, we then adopt active sampling strategies [56] such as in-batch hard negative pixels, to better discriminate the representations at a larger scale.
We further address the two other challenges: consistency and diversity. The success of the common CL theme is mainly attributed to invariant mapping [36] and instance discrimination [18,91]. Starting from these two key aspects, we try to further improve the segmentation quality. More specifically, we suggest that consistency to transformation (equivariance) is an effective strategy to establish the invariances (i.e., anatomical features and shape variance) to various image transformations. Furthermore, we investigate two ways to include diversity-promoting views in sample generation. First, we incorporate a memory buffer to alleviate the demand for large batch size, enabling much more efficient training without inhibiting segmentation quality. Second, we leverage stronger augmentations and nearest neighbors to mine views as positive views for more semantic similar contexts.
Extensive experiments are conducted on a variety of datasets and the latest CL frameworks (i.e., Mo-Cov2 [19], kNN-MoCo [84], SimCLR [18], BYOL [35], and ISD [81]), which consistently demonstrate the effectiveness of our proposed MONA. For example, our MONA establishes the new state-of-the-art performance, compared to both the state-of-the-art semi-supervised and fullysupervised approaches with 10% label ratio. Moreover, we present a systematic evaluation for analyzing why our approach performs so well and how different factors contribute to the final performance (See Section 4.2). We also study whether these principle can effectively complement each CL frameworks (See Appendix C). We hope our findings will provide useful insights on medical image segmentation to other researchers.
To summarise, our contributions are as follows: ❶ we carefully examine the problem of semi-supervised 2D medical image segmentation with extremely limited labels, and identify the three principles to address such challenging tasks; ❷ we construct a set of objectives, which significantly improves the segmentation quality, both long-tail class distribution and anatomical features; ❸ we analyse several critical components of our method and conduct thorough ablation studies to validate their necessity; ❹ with the combina-  tion of different components, we establish state-of-the-art, for all the challenging three benchmarks.

Related work
Medical Image Segmentation. Medical image segmentation aims to assign a class label to each pixel in an image, and plays a major role in real-world applications, such as assisting the radiologists for better disease diagnosis and reduced cost. With sufficient annotated training data, significant progress has been achieved with the introduction of Fully convolutional networks (FCN) [57] and UNet [72]. Follow-up works can be categorized into two main directions. One direction is to improve modern segmentation network design. Many CNN-based [39,78] and Transformerlike [26,85] model variants [1, 10, 15-17, 25, 37, 43, 61, 65, 82, 89, 90, 95, 98-101] have been proposed since then. For example, some works [16,17,24] proposed to use dilated/atrous/deformable convolutions with larger receptive fields for more dense anatomical features. Other works [10,15,37,82,95,101] include Transformer blocks to capture more long-range information, achieving the impressive performance. A parallel direction is to select proper optimization strategies, by designing loss functions to learn meaningful representations [55,77,96]. However, those methods assume access to a large, labeled dataset. This restrictive assumption makes it challenging to deploy in most real-world clinical practices. In contrast, our MONA is more robust as it leverages only a few labeled data and large quantities of unlabeled one in the learning stage. Semi-Supervised Learning (SSL). The goal in robust SSL is to improve the medical segmentation performance by taking advantage of large amounts of unlabelled data during training. It can be roughly categorized into three groups: (1) self-training by generating unreliable pseudolabels for performance gains, such as pseudo-label estimation [4,20,27,49], model uncertainty [9,11,33,45,60,63,104,106], confidence estimation [7,31,48], and noisy student [93]; (2) consistency regularization [8,22,28,29,111] by integrating consistency corresponding to different transformation, such as pi-model [74], co-training [70,112], and mean-teacher [53,71,80]; (3) other training strategies such as adversarial training [51,64,83,[107][108][109] and entropy minimization [34]. In contrast to these works, we do not explore more advanced pseudo-labelling strategy to learn spatially structured representations. In this work, we are the first to explore a novel direction for discovering distinctive and semantically consistent anatomical features without image-level or region-level labels. Further, we expect that our findings can be relevant for other medical image segmentation frameworks.
Contrastive Learning. CL has recently emerged as a promising paradigm for medical image segmentation via exploiting abundant unlabeled data, leading to state-of-theart results [13,14,42,92,94,102,103,105]. The high-level idea of CL is to pull closer the different augmented views of the same instance but pushes apart all the other instances away. Intuitively, differently augmented views of the same image are considered positives, while all the other images serve as negatives. The major difference between different CL-based frameworks lies in the augmentation strategies to obtain positives and negatives. A few very recent studies [44,46] confirm the superiority of CL of addressing imbalance issues in image classification. Moreover, existing CL frameworks [13,42,102] mainly focus on the in-stance level discrimination (i.e., different augmented views of the same instance should have similar anatomical features or clustered around the class weights). However, we argue that not all negative samples equally matter, and the above issues have not been explored from the perspective of medical image segmentation, considering the class distributions in the medical image are perspectives diverse and always exhibit long tails [32,73]. Inspired by the aforementioned, we address these two issues in medical image segmentation -two appealing perspectives that still remain under-explored.

.1. Framework
Overview. We introduce our contrastive learning framework (See Figure 2), which includes (1) relational semisupervised pre-training, and (2) anatomical contrastive reconstruction fine-tuning. The key idea is to seek diverse yet semantically consistent views whose anatomical features are homogeneous within the same class type, while distinctive for different class types in imbalanced and unlabeled clinical scenarios. In this paper, our pre-training stage is built upon ISD [81] -a competitive framework for image classification. The main differences between ISD and MONA are: ❶ MONA is more tailored to medical image segmentation, i.e., considering the dense nature of this problem both in global and local manners, and can generalize well to those long-tail scenarios; ❷ we directly work at the pixel-level local representations by allowing for the correspondence between anatomical features; ❸ we contrast the query image with other random images instead of only with different augmented views of the same query images. We believe our method can be formulated as the relaxed CL in both global and local manners. Such global and local relaxations enable the model to prioritize the most informative demonstrations among the entire dataset; ❹ our principles are expected to apply to other CL frameworks (i.e., Mo-Cov2 [19], kNN-MoCo [84], SimCLR [18], BYOL [35], and ISD [81]). More detailed analysis can be found in the Appendix C. Pre-training preliminary. Let (X, Y ) be our dataset, including training images x ∈ X and their corresponding Cclass segmentation labels y ∈ Y , where X is composed of N labeled and M unlabeled slices. Note that, for brevity, y can be either sampled from Y or pseudo-labels. The student and teacher networks F, parameterized by weights θ and ξ, each consist of a encoder E and a decoder D (i.e., UNet [72]). Concretely, given a sample s from our unlabeled dataset, we have two ways to generate views: (1) we formulate augmented views (i.e., x, x ′ ) through two different augmentation chains; and (2) we create d mined views (i.e., x r,i ) by randomly selecting from the unlabeled dataset followed by additional augmentation. 1 We then fed the augmented views to both F θ and F ξ , and the mined views to F ξ . In contrast to [81], we propose to perform instance discrimination in both global latent and local output feature spaces. 2 Specifically, the encoders generate global fea- , and z r,g = E ξ (x r ), which are then fed into the nonlinear projection heads to obtain v g = h θ (z g ), v ′ g = h ξ (z ′ g ), and w g = h ξ (z r,g ). The augmented embeddings from the student network are further projected into secondary space, i.e., u g = h ′ θ (v g ). We calculate similarities across mined views and augmented views from the student and teacher in both global and local manners. Then a softmax function is applied to process the calculated similarities, which models the relationship distributions: where τ θ and τ ξ are different temperature parameters, and sim(·, ·) denotes cosine similarity. The unsupervised instance discrimination loss (i.e., Kullback-Leibler divergence KL) can be defined as: The parameters ξ of F ξ is updated as: ξ = tξ + (1 − t)θ with t = 0.99 as a momentum hyperparameter. In our pretraining stage, the total loss is the sum of global and local instance discrimination loss L inst (on pseudo-labels), and supervised segmentation loss L sup (i.e., equal combination of dice loss and cross-entropy loss on ground-truth labels): L global inst + L local inst + L sup . Principles. As shown in Figure 2, the principles behind MONA (i.e., the second anatomical contrastive reconstruction stage) aim to ensure tailness, consistency, and diversity. Concretely, tailness is for actively sampling more tail class hard pixels; consistency ensures the feature invariances; and diversity further encourages to discover more anatomical features in different images.

Anatomical Contrastive Reconstruction
Tailness. Motivated by the observations (Figure 1), our primary cue is that medical images naturally exhibit an imbalanced or long-tailed class distribution, wherein many class labels are associated with only a few pixels. To generalize well on such imbalanced setting, we propose to use anatomical contrastive formulation (ACF) (See Figure 3).  Here we additionally attach the representation heads to fuse the multi-scale features with the feature pyramid network (FPN) [54] structure and generate the m-dimensional representations with consecutive convolutional layers. The high-level idea is that the features should be very similar among the same class type, while very dissimilar across different class types. Particularly for long-tail medical data, a naïve application of this idea would require substantially computational resources proportional to the square of the number of pixels within the dataset, and naturally overemphasize the anatomy-rich head classes and leaves the tail classes under-learned in learning invariances, both of which suffer performance drops.
To this end, we address this issue by actively sampling a set of pixel-level anchor representations r q ∈ R c q (queries), pulling them closer to the class-averaged mean of representations r c,+ k within this class c (positive keys), and pushing away from representations r − k ∈ R c k from other classes (negative keys). Formally, the contrastive loss is defined as: 3) where C denotes a set of all available classes for each minibatch, and τ is a temperature hyperparameter. Suppose A is a collection including all pixel coordinates within x, these representations are: We then note that CL might benefit more, where the instance discrimination task is achieved by incorporating  more positive and negative pairs. However, naively unrolling CL to this setting is impractical since it requires extra memory overheads that grow proportionally with the amount of instance discrimination tasks. To this end, we adopt a random set (i.e., the mini-batch) of other images. Intuitively, we would like to maximize the anatomical similarity between all the representations from the query class, and analogously minimize all other class representations. We then create a graph G to compute the pair-wise class relationship: G[p, q] = r p,+ k · r q,+ k , ∀p, q ∈ C, and p ̸ = q, where G ∈ R |C|×|C| . Here finding the accurate decision boundary can be formulated mathematically by normalizing the pair-wise relationships among all negative class representations via the softmax operator. To address the challenge in imbalanced medical image data, we define the pseudo-label (i.e., easy and hard queries) based on a defined threshold as follows: where y q is the c th -class pseudo-label corresponding to r q , and δ θ is the user-defined threshold. For further improvement in long-tail scenarios, we construct a class-aware memory bank [38] to store a fixed number of negative samples per class c. Consistency. The proposed ACF is designed to address imbalanced issues, but anatomical consistency remains to be weak in the long-tail medical image setting since medical segmentation should be robust to different tissue types which show different anatomical variations. As shown in Figure 4, we hence construct a random image transformation T and define the equivariance loss on both labeled and unlabeled data by measuring the feature consistency distance between each original segmentation map and the segmentation map generated from the transformed image:    ).
Here we define T on both the input image x and F θ (x), via the random transformations (i.e., affine, intensity, and photo-metric augmentations), since the model should learn to be robust and invariant to these transformations. Diversity. Oversampling too many images from the random set would create extra memory overhead, and more importantly, our finding also uncovers that a large number of random images might not necessarily help impose additional invariances between neighboring samples since redundant images might introduce additional noise during training (see the Appendix D). Therefore, we formulate our insight as an auxiliary loss that regularizes the representations -keeping the anatomical contrastive reconstruction task as the main force. In contrast to sampling from latent space [84], we perform in the output space (see Appendix D for more analysis). In practice, we first search for Knearest neighbors from the first-in-first-out (FIFO) memory bank [38], and then use the nearest neighbor loss L nn in a way that maximizing cosine similarity, to exploit the interinstance relationship.
Setup. The total loss L total is the sum of contrastive loss L contrast (on both ground-truth labels and pseudo-labels), equivariance loss L eqv (on both ground-truth labels and pseudo-labels), nearest neighbors loss L nn (on both groundtruth labels and pseudo-labels), unsupervised cross-entropy loss L unsup (on pseudo-labels) and supervised segmentation loss L sup (on ground-truth labels): L sup + λ 1 L contrast + λ 2 L eqv + λ 3 L unsup + λ 4 L nn . See the Appendix D for an ablation study of hyperparameters.

Experiments
In this section, we evaluate our proposed MONA on three popular medical image segmentation datasets under varying labeled ratio settings: the ACDC dataset [5], the LiTS dataset [6], and the MMWHS dataset [114] (See Ap-

Main Results
We show the effectiveness of our method under three different label ratios (i.e., 1%, 5%, 10%). We also compare MONA with various state-of-the-art SSL and fullysupervised methods on three datasets: ACDC [5], LiTS [6], MMWHS [114]. We choose 2D UNet [72] as backbone, and compare against SSL methods including UNet trained with full/limited supervisions (UNet-F/UNet-L), EM [87], CCT [67], DAN [107], URPC [59], DCT [70], ICT [86], MT [80], UAMT [104], CPS [20], SCS [42], GCL [13], and PLC [14]. We report quantitative comparisons on ACDC and LiTS in Table 1, and average all our results over three runs. (More results on MMWHS in Appendix B). ACDC. We benchmark performances on ACDC with respect to different amounts of labeled ratios (i.e., 1%, 5%, 10%). The following observations can be drawn: First, our proposed MONA significantly outperforms all other SSL methods under three different label ratios. Especially, with only extremely limited labeled data available (e.g., 1%), our method obtains massive gains of 82.6% and 2.03 in Dice and ASD (i.e., dramatically improving the performance from 59.4% to 82.6%). Second, our method achieves consistently improved performance, and performs better or on par with the fully-supervised approach under all three different label ratios. In particular, MONA with limited labeled training data available (e.g., 10%) -for the first time -surpasses the fully supervised counterparts. For example, 3 The PyTorch-like pseudo-code of our method can be found in the supplementary material. Codes will be available upon the publication. the best Dice score on ACDC rises from 89.9% to 90.7%. Third, as shown in Figure 5, we can see the clear advantage of MONA, where the anatomical boundaries of different tissues are clearly more pronounced such as RV and Myo regions. As seen, our method is capable of producing consistently sharp and accurate object boundaries across various challenge scenarios. LiTS. We then evaluate MONA on LiTS, using 1%, 5%, 10% labeled ratios. The results are summarized in Table 1 and Figure 6. The conclusions we can draw are highly consistent with the above ACDC case: First, at the different label ratios (i.e., 1%, 5%, 10%), MONA consistently outperforms all the other SSL methods, which again demonstrates the effectiveness of learning representations for the inter-class correlations and intra-class invariances under imbalanced class-distribution scenarios. In particular, our MONA, trained on a 1% labeled ratio (i.e., extremely limited labels), dramatically improves the previous best averaged Dice score from 59.3% to 64.1% by a large margin, and even performs on par with previous SSL methods using 10% labeled ratio. Second, the most impressive results come from MONA at 10% label ratio. To the best of our knowledge, this is the first time in the literature that SSL schemes trained at 10% label ratio outperform the fullysupervised model by 1.1% improvements in Dice (i.e., from 68.2% to 69.3%). Third, as shown in Figure 6, we observe that MONA is able to produce more accurate results compared to the previous best schemes.
Overall, we conclude that MONA provides robust performance on all the medical datasets we evaluated, exceeding that of the fully-supervised baseline, and outperforming all other SSL methods.

Ablation Study
In this subsection, we conduct comprehensive analyses to understand the inner workings of MONA on ACDC under 5% labeled ratio. Note that for reproducibility, we report the average performance of three independent runs with  (3) diversity. The results are in Table  2. As is shown, each key component makes a clear difference and leveraging all of them contributes to the remarkable performance improvements. This suggests the importance of learning meaningful representations for the interclass correlations and intra-class invariances within the entire dataset. The intuitions behind each concept are as follows: (1) Only tailness: many anatomy-rich head classes would be sampled; (2) Only consistency: it would lead to object collapsing due to the different anatomical variations; (3) Only diversity: oversampling too many negative samples often comes at the cost of performance degradation. By combining tailness, consistency, and diversity, our method confers a significant advantage at representation learning in imbalanced feature similarity, semantic consistency and anatomical diversity, which further highlights the superiority of our proposed MONA (More results in Appendix C). Effects of Different Augmentations. In addition to further improving the quality and stability in anatomical representation learning, we claim that MONA also gains robustness using augmentation strategies. For augmentation strategies, previous works [79,81,110] show that composing the weak augmentation strategy for the "pivot-to-target" model (i.e., trained with limited labeled data and a large number of unlabeled data) is helpful for anatomical representation learning since the standard contrastive strategy is too aggressive, intuitively leading to a "hard" task (i.e., introducing too many disturbances and yielding model collapses). Here we examine whether and how applying different data augmentations helps MONA. In this work, we implement the weak augmentation to the teacher's input as random rotation, random cropping, horizontal flipping, and strong augmentation to the student's input as random rotation, random cropping, horizontal flipping, random contrast, CutMix [30], brightness changes [69], morphological changes (diffeomorphic deformations). We summarize the results in Table 3, and list the following observations: (1) weak augmentations benefits more: composing the weak augmentation for the teacher model and strong augmentation for the student model significantly boosts the performance across two benchmark datasets. (2) same augmentation pairs do not make more gains: interestingly, applying same type of augmentation pairs does not lead to the best performance compared to different types of augmentation pairs. We postulate that composing different augmentations can be considered as a harder albeit more useful strategy for anatomical representation learning, making feature more generalizable. Extra Study. More investigations about (1) generalization across CL frameworks; (2) ablation on different training settings are in Appendix C and D.

Conclusion
In this paper, we have presented MONA, a semisupervised contrastive learning method for 2D medical image segmentation. We start from the observations that medical image data always exhibit a long-tail class distribution, and the same anatomical objects (i.e., liver regions for two people) are more similar to each other than different objects (e.g.liver and tumor regions). We further expand upon this idea by introducing anatomical contrastive formulation, as well as equivariance and invariances constraints. An extensive empirical study shows that we can formulate a generic set of perspectives that allows us to learn meaningful representations across different anatomical features, which can dramatically improve the segmentation quality and alleviate the training memory bottleneck. Extensive experiments on three datasets demonstrate the state-of-the-art performance of our proposed framework in the long-tailed medical data regimes with extremely limited labels. We believe our results contribute to a better understanding of medical image segmentation and point to new avenues for long-tailed medical image data in realistic clinical applications. The ACDC dataset was hosted in MICCAI 2017 ACDC challenge [5], which includes 200 3D cardiac cine MRI scans with expert annotations for three classes (i.e., left ventricle (LV), myocardium (Myo), and right ventricle (RV)). Following [58,88], we use 140, 20 and 40 3D scans for training, validation, and testing. Among the training scans including 1312 training slices, we follow the similar setting in [58,88] to define different label ratios, i.e., 1% (13 slices), 5% (65 slices), 10% (131 slices). For pre-processing, we adopt the similar setting in [13] by normalizing the intensity of each 3D scan (i.e., using min-max normalization) into [0, 1], and re-sampling all 2D scans and the corresponding segmentation maps into a fixed spatial resolution of 256 × 256 pixels.
The LiTS dataset was hosted in MICCAI 2017 Liver Tumor Segmentation Challenge [6], which includes 131 contrast-enhanced 3D abdominal CT volumes with expert annotations for two classes (i.e., liver and tumor). Following [75,76], we use the first 100 and the rest 31 3D scans for training, and testing. Among the training scans including 44102 training slices, we follow the similar setting in [75,76] to define different label ratios, i.e., 1% (441 slices), 5% (2205 slices), 10% (4410 slices). For pre-processing, we adopt the similar setting in [52] by truncating the intensity of each 3D scan into [−200, 250] HU for removing irrelevant and redundant details, normalizing each 3D scan into [0, 1], and re-sampling all 2D scans and the corresponding segmentation maps into a fixed spatial resolution of 256 × 256 pixels.

Representation Head
Ground-truth Labels Generated Labels G GT Figure 7. Representation head architecture.

A.2. Implementation Details.
We implement all the evaluated models using PyTorch library [68]. All the models are trained using Stochastic Gradient Descent (SGD) (i.e., initial learning rate = 0.01, momentum = 0.9, weight decay = 0.0001) with batch size of 6, and the initial learning rate is divided by 10 every 2500 iterations. All of our experiments are conducted on NVIDIA GeForce RTX 3090 GPUs. We first train our model with 100 epochs during the pre-training, and then retrain the model for 200 epochs during the fine-tuning. We set the temperature τ ξ , τ θ , τ as 0.01, 0.1, 0.5. The size of the memory bank is 36. During the pre-training, we follow the settings of ISD, including global projection head setting, and predictors with the 512-dimensional output embedding, and adopt the setting of local projection head in [42]. More specifically, given the predicted logits y ∈ R C×H×W , we create 36 different views (i.e., random crops at the same location) of y and y ′ with the fixed size 64 × 64, and then project all pixels into 512-dimensional output embedding space, and the output feature dimension of h ′ θ is also 512. An illustration of our representation head is presented in Figure 7. We then actively sample 256 query embeddings and 512 key embeddings for each mini-batch, and the con-  fidence threshold δ θ is set to 0.97. When fine-tuning we use an equally sized pool of candidates K = 5, as well as λ 1 = 0.01, λ 2 = 1.0, λ 3 = 1.0, and λ 4 = 1.0. For different augmentation strategies, we implement the weak augmentation to the teacher's input as random rotation, random cropping, horizontal flipping, and strong augmentation to the student's input as random rotation, random cropping, horizontal flipping, random contrast, CutMix [30], brightness changes [69], morphological changes (diffeomorphic deformations). We adopt two popular evaluation metrics: Dice coefficient (DSC) and Average Symmetric Surface Distance (ASD) for 3D segmentation results. Of note, the projection heads, the predictor, and the representation head are only used in training, and will be discarded during inference.

B. More Experiments Results -MMWHS
Lastly, we validate MONA on MMWHS, under 1%, 5%, 10% labeled ratios. The results are provided in Table 4 and Figure 8. Again, we found that MONA consistently outperforms all other SSL methods with a significant performance margin, and achieves the highest accuracy among all the SSL and fully supervised approaches under three labeled ratios. As is shown, MONA trained at the 1% labeled ratio significantly outperforms all other methods trained at the 1% labeled ratio, even over the 5% labeled ratio. Concretely, MONA trained at only 1% labeled ratio outperforms Table 5. Ablation study of different contrastive learning frameworks on ACDC under three labeled ratio settings (1%, 5%, 10%). We compare two settings: with or without fine-tuning on the segmentation performance (DSC[%]/ASD[mm]). We denote 'without fine-tuning" to only pretaining. On all three labeled settings, our methods (i.e., GLCon and MONA) significantly outperform all the state-of-the-art methods by a significant margin. All the experiments are run with three different random seeds. The best results are in bold. the second-best method (i.e., GLCon) both at the 1% and 5% labeled, yielding 12.3% and 0.4% gains in Dice. We also observe the similar patterns that, MONA performs better or on par with all the other methods at 10% labeled. Particularly, MONA trained at both 5% and 10% labeled ratio surpasses the fully-supervised scheme by 0.6% and 1.8% improvements in Dice, which again demonstrates the superiority of MONA in extremely limited labeled data regimes.

C. Generalization Study of Contrastive Learning Pre-training
As discussed in Section 3.1, our motivation comes from the observation that there are only very limited labeled data and a large amount of unlabeled data in real-world clinical practice. As the fully-supervised methods generally outperform all other SSL methods by clear margins, we postulate that leveraging massive unlabeled data usually introduces additional noise during training, leading to degraded segmentation quality. To address this challenge, "contrastive learning" is a straightforward way to leverage existing unlabeled data in the learning procedure. As supported in Section 4 and Appendix B, our findings have shown that MONA generalizes well across different benchmark datasets (i.e., ACDC, LiTS, MMWHS) with diverse labeled settings (i.e., 1%, 5%, 10%). In the following subsection, we further demonstrate that our proposed principles (i.e., tailness, consistency, diversity) are beneficial to various state-of-the-art CL-based frameworks (i.e., MoCov2 [19], kNN-MoCo [84], SimCLR [18], BYOL [35], and ISD [81]) with different label settings. More details about these three principles can be found in Section 3.2. Of note, to the best of our knowledge, MONA is the first SSL training scheme that consistently outperforms the fully-supervised method on diverse benchmark datasets with only 10% labeled ratio.
Training Details of Competing CL Methods. We identically follow the default setting in each CL framework [18,19,35,81,84] except the epochs number. We train each model in the semi-supervised setting. For labeled data, we follow the same training strategy in Section 3.1. As for unlabeled data, we strictly follow the default settings in each baseline. Specifically, for fair comparisons, we pre-train each CL baseline and our proposed CL pre-trained method (i.e., GLCon) for 100 epochs in all our experiments. Then we fine-tune each CL model with our proposed principles with the same setting, as provided in Appendix A. For kNN-MoCo [84], given the following ablation study we set the number of neighbors k as 5, and further compare different settings of k in kNN-MoCo [84] in the following subsection. All the experiments are run with three different random seeds.
Comparisons with CL-based Frameworks. Table 5 presents the comparisons between our proposed methods (i.e., GLCon and MONA) and various CL baselines. After analyzing these extensive results, we can draw several consistent observations. First, we can observe that our proposed GLCon achieves performance gains under all the labeled ratios, which not only demonstrates the effectiveness of our method, but also further verifies this argument using "global-local" strategy [13]. The average improvement in Dice obtained by GLCon could reach up to 2.53%, compared to the second best scores at different labeled ratios. Second, we can find that incorporating our proposed three principles significantly outperforms the CL baselines without fine-tuning, across all frameworks and different labeled ratios. These experimental findings suggest that our pro- Table 6. Ablation study of different principles across different contrastive learning frameworks under various labeled ratio settings (1%, 5%, 10%). Experiments are conducted on ACDC using UNet [72] as the backbone with three independent runs. Here we report the segmentation performance in terms of DSC[%] and ASD [mm]. On all three labeled settings, incorporating our methods (i.e., tailness, consistency, and diversity) consistently achieve superior model robustness gains across different state-of-the-art CL frameworks.  (c) Figure 9. Effects of k-nearest neighbour in global feature space, mined view-set size, and mined view patch size. We report Dice and ASD of GLCon on the ACDC dataset at the 5% labeled ratio. All the experiments are run with three different random seeds.
formance of MoCov2, kNN-MoCo, SimCLR, BYOL, ISD, and our GLCon by 39.1%, 38.5%, 40.9%, 41.2%, 34.3%, 33.3%, respectively. The ACDC dataset is a popular multiclass medical image segmentation dataset, with massive imbalanced or long-tailed class distribution cases. The imbalanced or long-tailed class distribution gap could result in the vanilla models overfitting to the head class, and generalizing very poorly to the tail class. With the addition of under-sampling the head classes, the principle -tailness -can be deemed as the prominent strategy to yield better generalization and segmentation performance of the models across different labeled ratios. Similar results are found under 5% and 10% labeled ratios. Third, over a wide range of labeled ratios, MONA can establish the new state-of-theart performance bar for semi-supervised 2D medical image segmentation. Particularly, MONA -for the first timeboots the segmentation performance with 10% labeled ratio over the fully-supervised method while significantly outperforming all the other semi-supervised methods by a large margin. In summary, our proposed methods (i.e., GLCon and MONA) obtain remarkable performance on all labeled settings. The results verify the superiority of our proposed three principles (i.e., tailness, consistency, diversity) jointly, which makes the model well generalize to different labeled settings, and can be easily and seamlessly plugged into all other CL frameworks [18,19,35,81,84] adopting the twobranch design, demonstrating that these concepts consis-tently help the model yield extra performance boosts for them all.

Generalization Across CL Frameworks
As demonstrated in Table 6, incorporating tailness, consistency, and diversity have obviously superior performance boosts, which is aligned with consistent observations with Section 4.2 can be drawn. This suggests that these three principles can serve as desirable properties for medical image segmentation in both supervised and unsupervised settings.
Does k-nearest neighbour in global feature space help? Prior work suggests that the use of stronger augmentations and nearest neighbour can be the very effective tools in learning additional invariances [84]. That is, both the specific number of nearest neighbours and specific augmentation strategies are necessary to achieve superior performance. In this subsection, we study the relationship of k-nearest neighbour in global feature space and the behavior of our GLCon for the downstream medical image segmentation. Here we first follow the same augmentation strategies in [84] (More analysis on data augmentation can be found in Section 4.2), and then conduct ablation studies on how the choices of k-nearest neighbour can influence the performance of GLCon. Specifically, we run GLCon on the ACDC dataset at the 5% labeled ratio with a range of k ∈ {3, 5, 7, 10, 12}. Figure 9(a) shows the ablation study on k-nearest neighbour in global feature on the segmentation performance. As is shown, we find that GLCon Figure 10. Effects of hyperparameters λ1, λ2, λ3, λ4. We report Dice and ASD of MONA on the ACDC dataset at the 5% labeled ratio.
All the experiments are run with three different random seeds.
at k = 5, 7, 10 have almost identical performance (k = 5 has slightly better performance compared to other two settings), and all have superior performance compared to all others. In contrast, GLCon -through the use of randomly selected samples -is capable of finding diverse yet semantically consistent anatomical features from the entire dataset, which at the same time gives better segmentation performance. Ablation Study of Mined View-Set Size.
We then conduct ablation studies on how the mined view-set size in GLCon can influence the segmentation performance. We run GLCon on the ACDC dataset at 5% labeled ratio with a range of the mined view-set size ∈ {12, 18, 24, 30, 36, 42, 48}. The results are summarized in Figure 9(b). As is shown, we find that GLCon trained with view-set size 36 and 42 have similar or superior performance compared to all other settings, and our model with view-set size of 36 achieves the highest performance. Ablation Study of Mined View Size. Lastly, we study the influence of mined view size on the segmentation performance. Specifically, we run GLCon on the ACDC dataset at the 5% labeled ratio with a range of the mined view size ∈ {8, 16, 32, 64, 128}. Figure 9(c) shows the ablation study of mined view size on the segmentation performance. As is shown, we observe that GLCon trained with mined view size of 32 and 64 have similar segmentation abilities, and both achieve superior performance compared to other settings. Here the mined view size of 64 works the best for GLCon to yield the superior segmentation performance. Conclusion. Given the above ablation study, we set k, mined view-set size, patch size as 5, 36, 64 × 64 in our experiments, respectively. This can contribute to satisfactory segmentation performance.

D. Ablation Study of Anatomical Contrastive Reconstruction
In this section, we give a detailed analysis on the choice of the parameters in the anatomical contrastive reconstruction fine-tuning, and take a deeper look and understand how they contribute to the final segmentation performance. All the hyperparameters in training are the same across three benchmark datasets. All the experiments are run with three different random seeds. Ablation Study of Total Loss L total . Proper choices of hyperparameters in total loss L total (See Section 3.2) play a significant role in improving overall segmentation quality. We hence conduct the fine-grained analysis of the hyperparameters in L total . In practice, we fine-tune the models with three independent runs, and grid search to select multiple hyperparameters. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of different hyperparameters λ 1 ∈ {0.005, 0.001, 0.05, 0.01, 0.05, 0.1}, and λ 2 , λ 3 , λ 4 ∈ {0.1, 0.2, 0.5, 1.0, 2.0, 10.0}. We summarize the results in Figure 10, and take the best setting λ 1 = 0.01, λ 2 = 1.0, λ 3 = 1.0, λ 4 = 1.0.  Figure 11. Effects of confidence threshold δ θ , K-nearest neighbour constraint, and output embedding dimension. We report Dice and ASD of MONA on the ACDC dataset at the 5% labeled ratio. All the experiments are run with three different random seeds.
Ablation Study of Confidence Threshold δ θ . We then assess the influence of δ θ on the segmentation performance. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of the confidence threshold δ θ ∈ {0.85, 0.88, 0.91, 0.94, 0.97, 1.0}. Figure 11(a) shows the ablation study of δ θ on the segmentation performance. As we can see, MONA on δ θ = 0.97 has superior performance compared to other settings.
Ablation Study of K-Nearest Neighbour Constraint.
Next, we conduct ablation studies on how the choices of K in K-nearest neighbour constraint can influence the segmentation performance. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of the choices K ∈ {3, 5, 7, 10, 12}. Figure 11(b) shows the ablation study of K choices on the segmentation performance. As we can see, MONA on K = 5 achieves the best performance compared to other settings.
Ablation Study of Output Embedding Dimension. Finally, we study the influence of the output embedding dimension on the segmentation performance of MONA. Specifically, we run MONA on the ACDC dataset at the 5% labeled ratio with a range of output embedding dimension ∈ {64, 128, 256, 512, 768}. Figure 11(c) shows the ablation study of output embedding dimension on the segmentation performance. As we can see, MONA with output embedding dimension of 512, can be trained to outperform other settings. Conclusion. Given the above ablation study, we select λ 1 = 0.01, λ 2 = 1.0, λ 3 = 1.0, λ 4 = 1.0, δ θ = 0.97, K = 5, output embedding dimension = 512 in our experiments. This can provide the optimal segmentation performance across different labeled ratios.


extfieldsofstudy: Biology Medicine

text:
Stardust: improving spatial transcriptomics data analysis through space-aware modularity optimization-based clustering

Abstract Background Spatial transcriptomics (ST) combines stained tissue images with spatially resolved high-throughput RNA sequencing. The spatial transcriptomic analysis includes challenging tasks like clustering, where a partition among data points (spots) is defined by means of a similarity measure. Improving clustering results is a key factor as clustering affects subsequent downstream analysis. State-of-the-art approaches group data by taking into account transcriptional similarity and some by exploiting spatial information as well. However, it is not yet clear how much the spatial information combined with transcriptomics improves the clustering result. Results We propose a new clustering method, Stardust, that easily exploits the combination of space and transcriptomic information in the clustering procedure through a manual or fully automatic tuning of algorithm parameters. Moreover, a parameter-free version of the method is also provided where the spatial contribution depends dynamically on the expression distances distribution in the space. We evaluated the proposed methods results by analyzing ST data sets available on the 10x Genomics website and comparing clustering performances with state-of-the-art approaches by measuring the spots' stability in the clusters and their biological coherence. Stability is defined by the tendency of each point to remain clustered with the same neighbors when perturbations are applied. Conclusions Stardust is an easy-to-use methodology allowing to define how much spatial information should influence clustering on different tissues and achieving more stable results than state-of-the-art approaches.

This work presents a new clustering method, Stardust, that has the potential to improve stability of clustering results against parameter changing. Stardust can assess the contribution to the clustering result by spatial information relative to gene expression information. Stardust appears to performs better than other methods in the two metrics used in this paper, stability and coefficient of variation. The essence of the method is the use of a spatial transcriptomics (ST) distance matrix as a simple linear combination of physical distance (S) and transcriptional distance (T) matrices. A weight factor is used for the S matrix to control and evaluate the contribution of the spatial information. The effort for evaluating multiple parameters and comparing with several latest methods and across a number of public spatial datasets is a highlight of the work. The authors also made the code available. Major comments: -The concept of combining spatial location and gene expression is not new and has been applied in most spatial clustering methods. It is not clear what are the new additions to current available methods, except for a feature to weigh the contribution of spatial components to clustering results. -The approach to assess the contribution of spatial information, by varying the weight factor from 0 to 1 is rather simple, because the contribution can be nonlinear and vary between spots/cells (e.g. spatial distance becomes more important for spots/cells that are nearer to each other; some genes are more spatially variable than the others; applying one weight factors for all genes and all spots would miss these variation sources) -The 5 weight factors 0, 0.25, 0.50, 0.75, and 1 were used. However, this range of parameters provided too few data points to assess the impact of spatial factor. As seen in figures, the 5 data points donot strongly suggest a point where the spatial contribution is maximum/minimum due to large fluctuation of values in the y-axis. -Although two performance metrics are used (stability and variation), there needs to be an additional metric about how the clustering results represent biological ground truth cell type composition or tissue architecture (for example, by comparing to pathological annotation). Consequently, it is unclear if the stardust results are closer to the biological ground truth or not.
-Stardust was tested on multiple 10x Visium datasets, but different types of spatial transcriptomics data like seqFISH, Slideseq, MERFISH, ect. are also common. Extended assessment of potential applications to other technologies would be useful. Minor comments: -The paragraphs and figure legends in the Result section are repetitive.
-The result section is descriptive and there is no Discussion section.

Level of Interest
Please indicate how interesting you found the manuscript: Choose an item.

Quality of Written English
Please indicate the quality of language in the manuscript: Choose an item.

Declaration of Competing Interests
Please complete a declaration of competing interests, considering the following questions:  Have you in the past five years received reimbursements, fees, funding, or salary from an organisation that may in any way gain or lose financially from the publication of this manuscript, either now or in the future?
 Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this manuscript, either now or in the future?
 Do you hold or are you currently applying for any patents relating to the content of the manuscript?
 Have you received reimbursements, fees, funding, or salary from an organization that holds or has applied for patents relating to the content of the manuscript?
 Do you have any other financial competing interests?  Do you have any non-financial competing interests in relation to this paper?
If you can answer no to all of the above, write 'I declare that I have no competing interests' below. If your reply is yes to any, please give details below.
I declare that I have no competing interests I agree to the open peer review policy of the journal. I understand that my name will be included on my report to the authors and, if the manuscript is accepted for publication, my named report including any attachments I upload will be posted on the website along with the authors' responses. I agree for my report to be made available under an Open Access Creative Commons CC-BY license (http://creativecommons.org/licenses/by/4.0/). I understand that any comments which I do not wish to be included in my named report can be included as confidential comments to the editors, which will not be published.
Choose an item.
To further support our reviewers, we have joined with Publons, where you can gain additional credit to further highlight your hard work (see: https://publons.com/journal/530/gigascience). On publication of this paper, your review will be automatically added to Publons, you can then choose whether or not to claim your Publons credit. I understand this statement.
Yes Choose an item.


Lightweight and Progressively-Scalable Networks for Semantic Segmentation

Multi-scale learning frameworks have been regarded as a capable class of models to boost semantic segmentation. The problem nevertheless is not trivial especially for the real-world deployments, which often demand high efficiency in inference latency. In this paper, we thoroughly analyze the design of convolutional blocks (the type of convolutions and the number of channels in convolutions), and the ways of interactions across multiple scales, all from lightweight standpoint for semantic segmentation. With such in-depth comparisons, we conclude three principles, and accordingly devise Lightweight and Progressively-Scalable Networks (LPS-Net) that novelly expands the network complexity in a greedy manner. Technically, LPS-Net first capitalizes on the principles to build a tiny network. Then, LPS-Net progressively scales the tiny network to larger ones by expanding a single dimension (the number of convolutional blocks, the number of channels, or the input resolution) at one time to meet the best speed/accuracy tradeoff. Extensive experiments conducted on three datasets consistently demonstrate the superiority of LPS-Net over several efficient semantic segmentation methods. More remarkably, our LPS-Net achieves 73.4% mIoU on Cityscapes test set, with the speed of 413.5FPS on an NVIDIA GTX 1080Ti, leading to a performance improvement by 1.5% and a 65% speed-up against the state-of-the-art STDC. Code is available at \url{https://github.com/YihengZhang-CV/LPS-Net}.

Introduction
Semantic segmentation is to assign semantic labels to every pixel of an image or a video frame. With the development of deep neural networks, the stateof-the-art networks have successfully pushed the limits of semantic segmentation with remarkable performance improvements. For example, DeepLabV3+  and Hierarchical Multi-Scale Attention (Tao et al., 2020) achieve 82.1% and 85.4% mIoU on Cityscapes test set, which are almost saturated on that dataset. The recipe behind these successes originates from multi-scale learning. In the literature, the recent advances involve utilization of multi-scale learning for semantic segmentation along three different dimensions: U-shape (Chen et al., 2020;Peng et al., 2017), pyramid pooling Zhao et al., 2017), and multi-path framework (Chen et al., 2016;Tao et al., 2020). The U-shape structure hierarchically fuses the features to gradually increase the spatial resolution and naturally produce multi-scale features. The pyramid pooling methods delve into multi-scale information through executing spatial or atrous spatial pyramid pooling at multiple scales. Unlike the former two research schemes, the multi-path frameworks resize the input image to multiple resolutions or scales and feed each scale into an individual path of a deep  network. By doing so, the multi-path design places the input resolutions from high to low in parallel and explicitly maintains the high-resolution information rather than recovering from low-scale feature maps. As a result, the learnt features are potentially more capable of classifying and localizing each pixel.
We employ this elegant recipe of multi-path framework and further evolve such type of architectures with good accuracy/speed tradeoff for semantic segmentation. Our philosophies are from two perspectives: (1) lightweighting computational units for semantic segmentation, and (2) progressively scaling up the network while balancing accuracy and inference latency. We propose to explore the first problem by probing the basic unit of convolutional blocks, including the type of convolutions and the number of channels in convolutions, on the basis of several uniqueness (e.g., large feature maps, thin channel widths) in efficient semantic segmentation. Moreover, we further elaborate different ways of interaction across multiple paths with respect to the accuracy/speed tradeoff. Based on these lightweight practice, we build a tiny model, and then progressively expand the tiny model along multiple possible dimensions and select a single dimension that achieves the best tradeoff in each step to alleviate the second issue of accuracy/speed balance.
To materialize the idea, we present Lightweight and Progressively-Scalable Networks (LPS-Net) for efficient semantic segmentation. Specifically, LPS-Net bases the multi-path design upon the low-latency regime. Each path takes the resized image as the input to an individual network, which consists of stacked convolutional blocks. The networks across paths share the same structure but have independent learnable weights. The outputs from all the paths are aggregated to produce the score maps, which are upsampled via bilinear upsampling for pixel-level predictions. In an effort to achieve a lightweight and efficient architecture, we look into the basic unit of convolutional blocks and empirically suggest to purely use 3×3 convolutions with 2 n -divisible channel widths. Furthermore, we capitalize on a simple yet effective way of bilinear interpolation to encourage mutual exchange and interactions between paths. With these practical guidelines, LPS-Net first builds a tiny network and then scales the tiny network to a family of larger ones in a progressive manner. Technically, LPS-Net executes the growth of network complexity through expanding a single dimension of the number of convolutional blocks, the number of channels, or the input resolution at one time, as depicted in Figure 1. In our case, LPS-Net ensures a good balance between accuracy and inference speed during expansion, and shows the superiority over other manually-/auto-deigned models, as shown in Figure 2.
In summary, we have made the following contributions: (1) The lightweight design of convolutional blocks and the way of path interactions in multipath framework are shown capable of regarding as the practical principles for efficient semantic segmentation; (2) The exquisitely devised LPS-Net is shown able to progressively expand the network complexity while striking the right accuracy-efficiency tradeoff; (3) LPS-Net has been properly verified through extensive experiments over three datasets, and superior capability is observed on both NVIDIA GPUs and embedded devices in our experiments.

Related Works
Semantic Segmentation. With the success of convolutional neural networks (CNNs), FCN (Long et al., 2015) which deploys CNN in a fully convolutional manner enables dense semantic prediction and endto-end training for semantic segmentation. Following FCN, researchers propose various techniques, which achieve remarkable performances. For example, in order to extract multi-scale context information which is known to be critical for pixel labeling tasks (Chen et al., 2016;Ladický et al., 2009), PSPNet (Zhao et al., 2017) applies spatial pyramid pooling and DeepLab (Chen et al., 2018a,b) utilizes parallel dilated convolutions with different rates. Yuan et al. (2021) propose to use object context represented as a dense relation matrix to emphasize the contribution of object information. Multi-layer feature aggregation Fu et al., 2019;Ghiasi and Fowlkes, 2016;Lin et al., 2019;Nirkin et al., 2021;Peng et al., 2017) is performed to recover the spatial information loss caused by spatial reduction in the network. Chen et al. (2016) and Tao et al. (2020) capitalize on the multi-path framework to improve the predictions. Such efforts are made to achieve high-quality segmentation but without taking inference latency into account.
Lightweight Networks. The real-world deployments often demand accurate and efficient networks. To accelerate the inference, there have been several techniques, e.g., pruning (Han et al., 2016), factorization (Szegedy et al., 2016), depthwise separable convolution (Chollet, 2017;Howard et al., 2017) and group convolution (Krizhevsky et al., 2012), being proposed in the literature. To facilitate model deployments and speed up the inference for semantic segmentation, recent works present to manually (Fan et al., 2021;Li et al., 2019aLi et al., , 2020bYu et al., 2021aYu et al., , 2018aZhao et al., 2018) or automatically (Chen et al., 2020;Li et al., 2020aLi et al., , 2019bLin et al., 2020;Zhang et al., 2019) design lightweight networks. ICNet  employs a cascade network structure to achieve realtime segmentation. BiSeNet (Yu et al., 2021a(Yu et al., , 2018a treats the spatial details and category semantics of the images separately to obtain a lightweight network. DFANet  designs a feature reuse method to incorporate multi-level context into encoded features. SFNet (Li et al., 2020b) boosts the segmentation by broadcasting high-level features to highresolution features with cheap operations. Fan et al. (2021) propose a short-term dense concatenate module to enrich features with scalable receptive field and multi-scale information in a lightweight network. Lite-HRNet (Yu et al., 2021b) utilizes the high-resolution design pattern of HRNet  and introduces conditional channel weighting to replace costly pointwise (1 × 1) convolutions in shuffle blocks . BiAlignNet  augments the BiSeNet by employing a gated flow alignment module to align features in a bidirectional way. To automate lightweight network design, CAS (Zhang et al., 2019) presents resource constraints to achieve an accuracy/computation tradeoff and GAS (Lin et al., 2020) further integrates a graph convolution network as a communication mechanism between different blocks. Li et al. (2019b) prunes the search space with a partial order assumption to search a balanced network. Chen et al. (2020) forms a search space integrating multiresolution branches and calibrates the balance between accuracy and latency by an additional regularization. AutoRTNet  jointly optimizes network depth, downsampling strategies, and the way of feature aggregation to obtain real-time segmentation networks.
Despite having such lightweight networks, the way to balance speed and accuracy for efficient semantic segmentation is not fully explored. Unlike the works like Yu et al., 2021b) which model different resolutions with standard Residual Conv blocks, our work aims at probing the basic design of Conv blocks, including the type of convolutions and the number of channels, particularly for efficient semantic segmentation. Instead of pre-fixing two-path design in networks like BiSeNet (Yu et al., 2021a(Yu et al., , 2018a and BiAlignNet , we exquisitely devise a progressive expansion paradigm to scale up the network. Such expansion dynamically balances the depth, width, resolution, and the number of paths of the networks to strike the right accuracy-efficiency tradeoff.
Summary. Our work focuses on developing lightweight and scalable networks for efficient semantic segmentation upon the low-latency regime. The proposal of LPS-Net contributes by studying not only the practical design of convolutional blocks and the way of path interactions in multi-path framework for semantic segmentation, but also how a scalable multipath network can be nicely expanded to meet the right accuracy/speed balance.

LPS-Net
We proceed to present our core proposal, i.e., the Lightweight and Progressively-Scalable Networks (LPS-Net). Specifically, we first introduce the macro architecture in LPS-Net that employs the multi-path recipe. Then, three design principles are presented to upgrade this architecture from lightweight standpoint, including the type of convolutions, the number of channels in  Fig. 3: An overview of the design space in our proposed LPS-Net. The red block and blue block represent the convolutional block with stride=2 and stride=1, respectively. The size of output feature map is given for each stage. Please note that the 3-path network here is an example to depict the macro architecture, and the number of paths in LPS-Net is dynamically determined by the parameter R = {r i | N i=1 } which is optimized for the right tradeoff along the network expansion.
convolution, and the way of interaction across multiple paths. Based on these practical guidelines, a family of scalable LPS-Net is devised in a progressive manner by expanding a single dimension at one time to seek the best speed/accuracy tradeoff.
3.1 Macro Architecture Figure 3 depicts an overview of the macro architecture in our LPS-Net. The macro architecture is basically constructed in the multi-path design, which resizes the input image to multiple scales and feeds each scale into an individual path. Formally, given the input image with resolution H × W , LPS-Net with N paths (e.g., N = 3 in Figure 3) resizes the image to r i H × r i W , which acts as the input of the i-th path. Here r i denotes the scaling ratio. Each path is implemented as a stack of convolutional blocks in five stages, and the j-th stage contains B j blocks. The number of channels (i.e., channel width) are maintained within each stage. Similar to , the first convolutional block in Stage 1∼4 reduces the spatial dimension by a factor of two. By doing so, the resolution of output feature map from the last stage is riH 16 × riW 16 . Moreover, we place interaction modules at the end of Stage 3∼5, aiming to encourage the mutual exchange and interactions between paths. The outputs of all paths are aggregated and fed into a segmentation head to produce the score maps with num classes channels. Finally, we perform bilinear upsampling over the score maps, yielding the outputs with resolution H × W that exactly matches the input resolution. Please note that the scaling ratios of paths R = {r i | N i=1 }, the repeated number of convolutional blocks B = {B j | 5 j=1 }, and the number of channels C = {C j | 5 j=1 } in this macro architecture can be flexibly set to make the network structure tailored to the target inference time and adjust the scalability of LPS-Net.

Convolutional Block
Convolutional block is the basic computing unit in Convolutional Neural Networks. The stack of convolutional blocks naturally consumes a high percentage of inference time throughout the entire network. Therefore, lightweighting convolutional block is essential for an efficient network. In this section, we delve into the design of basic computational block in LPS-Net along two dimensions: 1) the type of convolutional block, and 2) the number of channels.
The Type of Convolutional Block. In pursuit of less computational complexity, a series of innovations have been proposed to remould convolutional blocks (Han et al., 2020;He et al., 2016;Howard et al., 2017;Zhang et al., 2018). In these works, the amount of floating-point operations (FLOPs) is often utilized as the measurement of computational complexity, which guides the design of lightweight network for image recognition. However, FLOPs ignores the memory access cost (MAC) and the degree of parallelism that  substantially impact the inference latency of networks, thereby resulting in the discrepancy between FLOPs and actual latency. More importantly, the extension of lightweight designs from image recognition to semantic segmentation is not trivial due to several uniqueness in efficient semantic segmentation (e.g., larger feature maps and thin channel widths). To this end, we perform a comparison between the standard convolution and the widely-adopted depthwise separable convolution (SepConv) (Chollet, 2017;Howard et al., 2017;Tan and Le, 2019) in Figure 4. Here we set the shape of input feature map as 32 × 128 × 128, which is the most common shape in lightweight networks for semantic segmentation. The latency is evaluated by executing the block on a PC (i7-8700K/16GB RAM) with an 1070Ti GPU. The FLOPs-efficiency, which denotes the number of floating-point operations that is processed in per unit of run time, is utilized as the efficiency metric. As shown in Figure 4, the FLOPs-efficiency of standard convolution (3987 MFLOPs/ms) is about 10× more than that of SepConv (357 MFLOPs/ms). Accordingly, we utilize the standard convolution as the building block in LPS-Net.
The Number of Channels. The channel width is another important factor that influences the FLOPsefficiency of convolutions. Theoretically, the computational complexity necessitates quadratic progression with the increase of channel width. However, due to the highly-optimized software (e.g., BLAS, cuDNN) and hardware (e.g., CUDA Core, Tensor Core) in modern libraries/devices, the relation between defacto inference time and channel width in practice becomes fuzzy. To analyze this relation, we compare FLOPs and latency of 3 × 3 convolutions with different channel widths in Figure 5. As expected, FLOPs grows quadratically when increasing the channel width from 1 to 17 (see the blue curve in Figure 5(a)). Meanwhile, the latency only linearly grows and reaches plateaus at around 8 and 16 channel widths (see the red curve in Figure 5  setups of channel width with high FLOPs-efficiency are regarded as the "sweet spots" for convolutions. Similarly, high FLOPs-efficiencies are attained at 32 and 64 channel widths in Figure 5(b). These sweet points are mainly attributed to the highly parallelizable implementation of convolutions with 2 n -divisible channel width. As a result, we make the channel width of the convolution in LPS-Net to be 2 n -divisible with n as large as possible.

Multi-Path Interaction
In an effort to elegantly diffuse the complementary information across multiple scales, we design the multipath interaction module that encourages mutual exchange and interactions between pathes in our LPS-Net. Concretely, we define the interaction module I h , x out l } denote the input pair and output pair, respectively. The input pair consists of a high-resolution feature map x in h ∈ R C×H h ×W h and a low-resolution feature map x in l ∈ R C×H l ×W l , where H h ≥ H l and W h ≥ W l . The resolution of each path remains unchanged during the interaction.
Here, we study six variants of interaction modules which can be grouped into three categories ( Figure 6). For the first category (Direct-A/B), the transformed low-resolution features are directly integrated into the high-resolution ones through element-wise summation or channel-wise concatenation. Rather than simply combining the features from two paths, the second category employs attention mechanism to boost the multi-path interaction, namely Attention-A/B. Specifically, in Attention-A, an attention map Att(x in l ) is first calculated from x in l , and we apply it over the transformed x in h and x in l before feature aggregation: where F and U denote the convolutional transformation and spatial upsampling, respectively. Attention-B further upgrades the simple attention in a bi-direction manner by integrating each path with the attention derived from another path. In practice, the above four types of interaction modules are computationally expensive due to the heavy transformations and attentions. Thus, techniques (Chen et al., 2016;Tao et al., 2020;Yu et al., 2021a;Zhao et al., 2018) usually place them at the end of the multi-path networks, while leaving the path interactions at early stages unexploited. To alleviate this issue, two efficient interaction modules are devised to enable the interactions at early stages, namely Bilateral-A/B. Bilateral-A transforms low-resolution feature and injects it into the high-resolution one, and vice versa. Such symmetrical design treats each network path equally and the simple transformation triggers the interaction at early stages in an efficient fashion. Bilateral-B further eliminates the convolutional transformation in Bilateral-A, leading to the simplest interaction by solely applying spatial resizing and element-wise summation on input feature maps: where U and D denote upsampling and downsampling of the feature maps. The comparisons between all the six variants of interactions will be elaborated in the experiments, and we empirically verify that Bilateral-B achieves the best speed/accuracy tradeoff. Therefore, we adopt Bilateral-B as the default interaction module in our LPS-Net.

Scalable Architectures
Given the selected convolutional block and interaction module, a particular LPS-Net architecture can be for all δ ∈ ∆ do 14: end for 16: for all δ ∈ ∆ do 18: Expand N i−1 using δ with stepsize k δ . 21: P δ = P erf (N i δ ) 22: Train and evaluate N i δ for semantic segmentation. 23:

26: Select operation by comparing tradeoffs. 27:
Update: N i = N i−1 + k δ * · δ * , P = P δ * , L = L δ * 28: end while defined as N = {B, C, R}, where B, C, R are adjustable parameters described in Section 3.1. The network complexity is thus determined by these parameters from three dimensions. Depth dimension (B) is the number of blocks which determines the ability of the network to capture the high-level semantic information.
Width dimension (C), i.e., the number of channels at each stage, impacts the learning capacity of each convolution. Resolution dimension (R) represents the spatial granularity of each path. To balance the three dimensions when designing LPS-Net architecture, we start by building a tiny network, and then expand a single dimension at one time in a progressive manner.
Technically, we set the parameters of tiny network N 0 as B 0 = {1, 1, 1, 1, 1}, C 0 = {4, 8, 16, 32, 32}, and R 0 = {1/2, 0, 0}. Please note that the path with r = 0 is not included, and thus the tiny network only contains a single path. In this way, the tiny network is very efficient with only 0.38ms latency on a single GPU, and we further scale it to the heavier ones by multi-step expansion. All the candidates of expanding operations ∆ = ∆ B ∪ ∆ C ∪ ∆ R are defined in Table 1. When N = {B, C, R} is expanded by δ, δ will be added to N along its corresponding dimension and produce the expanded network (e.g. {B + δ, C, R} when δ ∈ ∆ B ). For each step, we greedily determine the executed expanding operation by comparing the tradeoff between speed and accuracy of each operation δ ∈ ∆. Specifically, for the i-th step, the target latency after expansion is first determined by the maximum latency across different expanding operations: where N i−1 denotes the architecture from the last step and Lat(·) measures the latency of the network. Then, a stepsize for each operation is calculated as: targeting for better aligning the resultant latency to L T after expanding different operations. Nevertheless, since the expansion only produces networks in discrete steps, the latencies of the expanded networks may exceed L T . If the selection of operation only depends on the performances of expanded networks, it may be biased to the operations which leads to a larger latency increase. To tackle this issue, the optimal expanding operation δ * is finally selected by maximizing the ratio of performance increase and latency increase after expanding at each step: where P erf (·) evaluates the performances of semantic segmentation. As such, starting from the tiny network N 0 , we iteratively perform the network expansion I times and finally derive a family of LPS-Net with different complexities. Algorithm 1 summarizes the expansion algorithm of LPS-Net.

Experiments
We empirically verify the merit of our LPS-Net by conducting a thorough evaluation of efficient semantic segmentation on Cityscapes (Cordts et al., 2016). Expressly, we first undertake the experiments for semantic segmentation to validate the lightweight designs of the convolutional blocks and the way of interactions across paths in LPS-Net with a pre-fixed structure. The second experiment examines the accuracy and latency tradeoffs along the progressive growth of network complexity of LPS-Net with regard to the number of convolutional blocks, the number of channels, and the input resolution. The third experiment compares three scalable levels of LPS-Net with several state-of-the-art fast semantic segmentation methods and demonstrates the better tradeoffs. Furthermore, we evaluate the transferability of the three versions of LPS-Net, which are learnt on Cityscapes, on BDD100K (Yu et al., 2018b) and CamVid (Brostow et al., 2008) datasets for semantic segmentation. Finally, we also migrate our LPS-Net to embedded devices and prove the efficacy on two different devices.

Datasets
The Cityscapes dataset contains 5,000 urban street images with high-quality pixel-level annotations of 19 classes. The image resolution is 1, 024 × 2, 048. All the images are split into three sets of 2,975, 500 and 1,525 for training, validation, and testing, respectively. For fair comparison, the additional set of 23,473 coarsely annotated images of this dataset is not utilized in our experiments. We train LPS-Net on the training set and evaluate the networks on the validation set. Following the standard protocol in (Fan et al., 2021;Lin et al., 2020;Zhang et al., 2019), LPS-Net is learnt on the training plus validation sets when submitting the model to online Cityscapes server and reporting the performance on the testing set.
To further examine the generalization of LPS-Net, we re-evaluate the networks on BDD100K (Yu et al., 2018b) and CamVid (Brostow et al., 2008) datasets. BDD100K is a recently released urban street dataset. For semantic segmentation, BDD100K includes 8,000 images being annotated with the consistent labels in Cityscapes. In between, 7,000 and 1,000 images are exploited for training and validation, respectively. The image resolution is 720 × 1, 280. CamVid consists of 5 video sequences with resolution 720×960, and is labeled at one frame per second with 11 semantic categories. 468 and 233 labeled frames in CamVid is utilized for training and testing, respectively.

Implementation Details
We implement our proposal on PyTorch (Paszke et al., 2019) platform and employ mini-batch Stochastic Gradient Descent (SGD) algorithm for model optimization.  (Ma et al., 2018), Inverted Residual (Sandler et al., 2018), Ghost Module (Han et al., 2020), and Residual/Bottleneck (He et al., 2016 In the training stage, we utilize the cross-entropy loss and train LPS-Net from scratch for 90K iterations, with a batch size of 16, momentum 0.9 and weight decay 0.0005. "Poly" policy with power 0.9 is adopted with initial learning rate 0.01. We use a crop size of 768×768 and exploit color-jittering, random horizontal flip and random scaling (0.5× ∼ 2.0×) to augment training data. Model re-parameterization (Ding et al., 2021) is also adopted for training. To further improve the capability of multi-path networks, we involve more powerful training strategies for the experiments in Section 4.5, including ImageNet (Russakovsky et al., 2015) pretraining, large crop size (768 × 1, 536), more training iterations (180K) and online hard element mining (Chen et al., 2020;Fan et al., 2021). Note that these strategies only enhance the network training and have no influence on model inference. Such training strategies employed for experiments on Cityscapes dataset are adjusted to train LPS-Net on BDD100K and CamVid datasets. For the experiments on BDD100K, we use a crop size of 720 × 1280 and the other strategies remain unchanged. For CamVid, we train the LPS-Net for 40K iterations, with a batch size of 12, an initial learning rate of 0.03, and a crop size 720 × 960. ImageNet pre-training and online hard element mining are also involved in experiments on BDD100K and CamVid.
In the inference stage, we input the image into our LPS-Net and perform semantic segmentation without any evaluation tricks (e.g., flipping, sliding-window testing and multi-scale inference), which in general improve the performance but at the expenses of extra latency. The Intersection over Union (IoU) per category and mean IoU over all the semantic categories are utilized as the performance metric. The inference latency of the network is measured by running networks with batch size of 1 on one NVIDIA GTX 1070Ti (Section 4.3 and Section 4.4), 1080Ti (Section 4.5) or two embedded devices (Section 4.6) with TensorRT in FP32 mode, unless otherwise stated.

Evaluation on Lightweight Designs
We first examine the lightweight designs of the convolutional blocks and the ways of interactions across paths in LPS-Net. We start the validation on the impact of the type of convolutional blocks. Specifically, we capitalize on seven convolutional blocks to build the corresponding multi-path networks. Following (Chen et al., 2016;Ding et al., 2021), B and R of all the networks are respectively set to {1, 2, 4, 7, 7} and {1, 1/4, 0}.
The channel widths C are set to {1c, 2c, 4c, 8c, 8c} for all the networks, where c is to adjust the inference latency of each network to approximate 5ms for fair comparisons. In order to purely verify the convolutional blocks irrespective of the interaction influence, we do not include any path interactions here and exploit channel-wise concatenation to aggregate the outputs from all paths of the multi-path network. Table 2 summarizes the mIoU performances, latencies, FLOPs, and FLOPs-efficiencies of the multi-path networks built on seven types of convolutional blocks. Similar to the observations in Section 3.2, 3×3 SepConv shows a very low FLOPs-efficiency. Conditioning on almost the same latency with 3×3 Conv, the utilization of 3 × 3 SepConv results in much fewer FLOPs, making the computational capability of the network weak. 3×3 SepConv is also inferior to 3 × 3 Conv in terms of mIoU performance. Furthermore, ShuffleNet Unit, Inverted Residual and Ghost Module are proven to be three very effective and lightweight convolutional blocks and all three blocks involve 3 × 3 SepConv. Inverted Residual and Ghost Module have benefited from the inverted residual structure with linear bottleneck and a series of linear transformations with cheap cost, respectively. They both greatly improve 3×3 SepConv, but the mIoU performances are still lower than that of 3 × 3 Conv. Residual contains a stack of two 3×3 convolutions with a shortcut connection and is extended to Bottleneck by stacking three convolutions (1 × 1, 3 × 3, 1 × 1) instead of two. The use of 1 × 1 convolution in Bottleneck leads to block fragmentation and affects its FLOPsefficiency. Compared to 3 × 3 Conv, an extra short Table 4: Network Parameters (Depth-B, Width-C and Resolution-R), latency, mIoU performance and expanding dimension of the network at each expansion step of LPS-Net.
Step B C R Latency (ms) mIoU (%) Dimension Name cut in Residual slightly lowers FLOPs-efficiency and increases the latency by ∼3%. Benefiting from the high FLOPs-efficiency and better tradeoff, 3×3 Conv exhibits the highest mIoU (66.2%) with the lowest latency (5.08ms). We further analyze the high FLOPsefficiency of 3×3 Conv from the perspective of devices. FLOPs-lightweight convolutions (e.g. Ghost Module and 3 × 3 SepConv) usually suffer from a low ratio of computing to memory operations (Gholami et al., 2018) and introduce extra overheads like kernel launching and synchronization. In contrast, 3×3 Conv is more friendly to devices and could be accelerated by the Winograd supported in modern computing libraries like cuDNN. As such, 3×3 Conv has high FLOPs-efficiency. Overall, 3×3 Conv achieves the best tradeoff and thus we exploit it as the convolutional block in LPS-Net.
Then, we study how the interactions across paths influence the accuracy-latency tradeoff of the multipath networks. In this comparison, the network parameters B and R are kept unchanged. We fix c = 16 and exploit 3 × 3 Conv as the convolutional block to construct the multi-path networks. Table 3 details the mIoU performance and latency when integrating different path interactions in multi-path networks. As expected, the results across six ways of interactions consistently indicate that using interaction exhibits better mIoU performance and higher latency against Baseline which does not involve path interaction. More specifically, the larger mIoU improvements are attained when employing the Bilateral-A/B interactions than the Direct-A/B and Attention-A/B interactions. Bilateral-B purely basing on bilinear interpolation leads to less increase of latency than Bilateral-A which uses two additional convolutional transformations. Among all the six interactions, Bilateral-B achieves the largest mIoU gain of 3.0% with the least extra latency of 0.29ms, and hence we utilize Bilateral-B as the path interaction in our LPS-Net.

Evaluation on Progressively-Scalable Scheme
Next, we analyze the accuracy-latency balance as the progressive expansion of LPS-Net proceeds. Figure 7 depicts the first 14 steps of the process on Cityscapes. Table 4 further details the parameters, latency, mIoU performance and expanding dimension of the network at each expansion step of LPS-Net in Figure 7. We initialize LPS-Net with a tiny structure and the network parameters are B 0 = {1, 1, 1, 1, 1}, C 0 = {4, 8, 16, 32, 32}, R 0 = {1/2, 0, 0}. The mIoU and latency of such start point is 24.1% and 0.38ms, respectively. As shown in the figure, expanding one  (Li et al., 2019a) 1024 × 1024 -71.3 100.0 TitanX DFANet B (Li et al., 2019a) 1024 × 1024 -67.1 120.0 TitanX DFANet A' (Li et al., 2019a) 512 × 1024 -70.3 160.0 TitanX Fast-SCNN (Poudel et al., 2019) 1024 × 1024 69.2 68.0 123.5 TitanXp MSFNet (Si et al., 2020) 512 × 1024 -71.3 117.0 2080Ti SFNet(DF1) (Li et al., 2020b) 1024 × 2048 -74.5 121.0 1080Ti TensorRT STDC1Seg-50 (Fan et al., 2021) 512 × 1024 72.2 71.9 250.4 1080Ti TensorRT STDC2Seg-50 (Fan et al., 2021) 512 certain dimension of Depth (the number of convolutional blocks), Width (the number of channels), or the input Resolution in LPS-Net always improves the mIoU performance. The fact that the selected dimension is various to strike the best mIoU performance-inference latency tradeoff at different steps necessitates the use of multiple dimensions for expansion. The first step is along the network Depth direction and significantly boosts up the mIoU to 41.4% with only 0.11ms additional latency. We speculate that this may be the result of feature enrichment by stacking more blocks particularly in the context of small channel width and low image resolution. In this case, deepening networks could readily enhance the network capability. The second and third steps further expand Width dimension, and the network after three steps achieves 57.8% mIoU with less than 1ms latency. The results indicate that the network tends to expand the Depth and Width dimensions especially at the very early steps. At the sixth step, the expansion proceeds along the Resolution direction and starts to develop a two-path networks. The next two consecutive steps also work on the factor of Resolution and increase R to {1, 1/4, 0}. The mIoU performance of the two-path networks at the eighth step reaches 69.5% and the latency is 5.17ms. That basically verifies the effectiveness of multi-path design to manage the accuracy-latency balance for semantic segmentation. The following steps continue the expansion along either the Width or Resolution dimension. Such an expansion process indicates some practical insights for efficient semantic segmentation. For example, stacking more blocks for tiny networks could effectively improve the segmentation with negligible extra latency. The multipath fusion is not always the best choice especially when the networks' inference latency is extremely constrained (e.g., ≤ 2ms). Compared to increasing channel width and enlarging input resolutions, fusing three or more paths is not optimal for efficient semantic segmentation.
To cover different levels of LPS-Net in complexity, we choose the networks at the seventh, eighth, and tenth steps as the small, medium, and large version of LPS-Net. We name these networks as LPS-Net-S, LPS-Net-M, and LPS-Net-L, respectively.

Comparisons with State-of-the-Art Methods
We compare our LPS-Net with several state-of-the-art fast semantic segmentation methods. To ensure the inference speed is comparable with the most recent works (Chen et al., 2020;Fan et al., 2021;Yu et al., 2021a), we report the inference FPS of LPS-Net on an NVIDIA GTX 1080Ti GPU card here. Table 5 summarizes the comparisons of mIoU performance and inference FPS on Cityscapes dataset. Note that only the models with FPS≥100 are included in the table. Overall, the results across validation and testing sets consistently indicate that LPS-Net exhibits better tradeoffs than other techniques including hand-crafted networks (e.g., (Fan et al., 2021;Li et al., 2019)) and auto-designed models (e.g., (Chen et al., 2020;Zhang et al., 2019)). In particular, the mIoU score of LPS-Net-S achieves 73.4% on test set, making the absolute improvement over the best competitor STDC1Seg-50 by 1.5%. Moreover, LPS-Net-S is faster than STDC1Seg-50 and the inference speed reaches 413.5 FPS, which is impressive. The large version of LPS-Net, i.e., LPS-Net-L, attains 78.2%/77.3% mIoU on validation/testing set at 151.8 FPS. It runs 49% faster than DDRNet23slim with comparable mIoU. This confirms the effectiveness of LPS-Net. We showcase the qualitative results of LPS-Net-S, LPS-Net-M, and LPS-Net-L in Figure 8. Similar to the observation in Table 5, the quality of prediction is consistently improved when the LPS-Net is incrementally expanded. For example, the prediction in the regions of pole (first row), car (second row), and wall (third row) is amended by expanding the complexity of LPS-Net.

LPS-Net-S on Embedded Devices
The real-world scenarios often demand the deployments on embedded devices. Here, we also test the small version of our LPS-Net, i.e., LPS-Net-S, on two devices of a single Nano and a single TX2, both of which are based on NVIDIA Jetson platform. Table 8 compares LPS-Net-S with DF1-Seg-d8 (Li et al., 2019b), FasterSeg (Chen et al., 2020), BiSeNetV2 (Yu et al., 2018a), and STDC1Seg-50 (Fan et al., 2021), regarding the mIoU performance on Cityscapes and inference  speed (FPS) on the two devices. Note that the inference speed is measured by using FP32 data precision to avoid performance degradation. Clearly, LPS-Net-S strikes the superior tradeoff and achieves 10.0/25.6 FPS on Nano/TX2 at resolution 1024×2048, which is 41%/50% faster than the best competitor STDC1Seg-50 (Fan et al., 2021). The results demonstrate the advantage of LPS-Net on embedded devices.

Conclusion and Discussion
We have presented Lightweight and Progressively-Scalable Networks (LPS-Net) which explores the economic design and progressively scales up the network for efficient semantic segmentation. Particularly, we analyze the basic convolutional block and the way of path interaction in multi-path framework which could affect the accuracy-latency tradeoff for semantic segmentation. Empirically, we suggest to use 3 × 3 Conv in convolutional blocks and bilinear interpolation to implement interactions across paths. Based on these guidelines, we first build a tiny network and then extends the tiny network to a series of larger ones through expanding a certain dimension of Width, Depth or Resolution at one time. Experiments conducted on three datasets, i.e., Cityscapes, CamVid and BDD100K, validate our proposal and analysis on both GPUs and embedded devices. More remarkably, our LPS-Net shows impressive accuracy-latency tradeoff on Cityscapes: 413.5 FPS on an NVIDIA GTX 1080Ti GPU or 25.6 FPS on an NVIDIA TX2, with mIoU of 73.4% on testing set.