extfieldsofstudy:
Computer Science
Medicine

s2fieldsofstudy:
Computer Science
Physics

text:
Supernovae Detection with Fully Convolutional One-Stage Framework

A series of sky surveys were launched in search of supernovae and generated a tremendous amount of data, which pushed astronomy into a new era of big data. However, it can be a disastrous burden to manually identify and report supernovae, because such data have huge quantity and sparse positives. While the traditional machine learning methods can be used to deal with such data, deep learning methods such as Convolutional Neural Networks demonstrate more powerful adaptability in this area. However, most data in the existing works are either simulated or without generality. How do the state-of-the-art object detection algorithms work on real supernova data is largely unknown, which greatly hinders the development of this field. Furthermore, the existing works of supernovae classification usually assume the input images are properly cropped with a single candidate located in the center, which is not true for our dataset. Besides, the performance of existing detection algorithms can still be improved for the supernovae detection task. To address these problems, we collected and organized all the known objectives of the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) and the Popular Supernova Project (PSP), resulting in two datasets, and then compared several detection algorithms on them. After that, the selected Fully Convolutional One-Stage (FCOS) method is used as the baseline and further improved with data augmentation, attention mechanism, and small object detection technique. Extensive experiments demonstrate the great performance enhancement of our detection algorithm with the new datasets.

Introduction
A series of sky surveys, such as the Dark Energy Surveys [1], the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) [2], the Large Synoptic Survey Telescope (LSST) [3] and so on, were launched in search of transient astronomical events that are astronomical sources or phenomena with limited duration such as supernovae. These sky surveys generated a tremendous amount of data, which pushed astronomy into a new era of big data. It is important to detect supernovae quickly and accurately. However, many data processing pipelines for these sky surveys rely strongly on human inspection. It can be a disastrous burden to manually identify and report supernovae, because such data have huge quantity and sparse positives. So there is an urgent need for a fast and automatic supernovae detection system.
There are some existing works leveraging machine learning methods in this area. Some researches dedicating to supernovae classification, that is, to differentiate supernovae or artifacts, have already reached relatively high performance. Du Buisson et al. [4] tested and compared many traditional machine learning methods for supernovae classification.
The Random Forest (RF) [4,5] algorithm performs best among traditional machine learning methods. With the development of deep learning methods such as Convolutional Neural Network (CNN), they show strong adaptability to deal with massive astronomical data. Cabrera-Vives et al. [6] proposed a CNN classifier that outperforms the Random Forest model. Another direction of this study is to analyze the variation of supernova spectral sequences. Again, deep learning methods represented by Recurrent Neural Networks (RNN) have achieved very good results in this task. Muthukrishna et al. [7] introduced the Real-time Automated Photometric Identification (RAPID) architecture and implemented fine-grade supernovae classification. Burke et al. [8] applied object detection to the astronomical source classification.
However, there are still some limitations of the existing works. A review article in Nature [9] argued that researchers are facing a challenge in the supernovae classification: Dataset is in the absence of statistically representative of the supernova. In fact, most data in the existing works either are simulated or lack generality. Furthermore, most of the existing works study supernovae classification, which assumes the input images are properly cropped with a single candidate located in the center and returns the label or class of the candidate. However, for images in some datasets, the candidates may not be centrally located, or there are multiple candidates in a single image. These cases can be regarded as supernovae detection tasks, which should return locations of detected supernovae. Besides, the performance of existing detection algorithms for general object detection tasks can still be improved for the supernovae detection task.
To address the above two issues, in our work we constructed two new datasets for supernovae detection, which consist of data from the Pan-STARRS and the Popular Supernova Project (PSP). The typical detection method is to compare the difference between the two images taken in the same azimuth but at different time. So all the image data we collected are difference images. The data from the Pan-STARRS have relatively high quality, while the data in the PSP are defective images from the public observations. Most of the PSP images have a relatively low signal-to-noise ratio (SNR), due to factors like bad weather, moonlight, equatorial mount failure, and so on. To detect supernovae in our datasets, we first compared several mainstream object detection algorithms and chose the best one, that is, the Fully Convolutional One-Stage (FCOS) algorithm [10], and then improved FCOS on this specific task.
The main goal of this paper is to develop an automatic supernovae detection framework with object detection techniques and thus accelerate the processing of observation data. Our contributions in this paper are summarised as follows:

•
We made a labeled dataset, which consists of 12,447 images from the Pan-STARRS with all supernovae labeled. • We made a labeled dataset, which consists of 716 images from the PSP with all supernovae labeled. Considering the amount of samples in the PSP dataset is too small, we also used the data augmentation technique on this dataset. • We compared several detection algorithms on both datasets. The FCOS method with better performance is used as the baseline method. In addition, the FCOS algorithm is improved with different techniques, such as data augmentation, attention mechanism, and increasing input size. To verify the challenge mentioned in the [9], we have tested the performance of model training for the hybrid datasets with different blending ratios.
The rest of this paper is structured as follows-the relevant existing works of supernovae classification, object detection, and attention mechanism are reviewed in Section 2. Then we introduce our datasets in Section 3. In Section 4, the supernovae detection framework and improvements are presented. In Section 5, we conduct extensive experiments to evaluate the detection methods on the two datasets. Some other methods are discussed in Section 6. Finally, we summarize our work in Section 7.

Supernovae Classification
In recent years, Machine Learning (ML) based algorithms have just been applied in supernovae classification, which is to differentiate types of supernovae. Du Buisson et al. [4] widely studied the performance of different ML algorithms on the 2nd and 3rd years of the Sloan Digital Sky Survey (SDSS)-II supernova survey. They extracted the features with Principal Component Analysis (PCA) and the normalized data are cropped by 31 × 31 pixels at the center. They tested and compared many algorithms on the processed data. The result shows that the Random Forest algorithm has the best performance in these classifiers. The Random Forest (RF) is an ensemble classifier. In their work, the optimal splitting points are selected from a random sample of features at every node, and then they split the data into different decision trees. Moreover, they studied the Cohen's kappa coefficient [11] between these algorithms and it turns out that these algorithms can at most reach moderate agreements.
Cabrera-Vives et al. [12] proposed a Convolutional Neural Network (CNN) classifier to classify different objects in images as transients or artifacts. They designed a neural network with 2 convolutional layers and 2 pooling layers, which are followed by 2 fully connected layers to resize the output into a 1 × 2 vector. Each element of the vector represents the possibility of transients or artifacts. They achieved higher performance than the RF model. Also based on CNN, Kimura et al. [13] proposed a deeper neural network with 3 convolutional layers combined with 5 channels for different frequency bands on a self-built dataset.
In 2017, Cabrera-Vives et al. [6] proposed a rotation-invariant CNN model which outperforms their former work. Their neural network has four branches. The candidates will be rotated in 90 • , 180 • , 270 • first and then four images will be fed into each branch. Each branch includes five convolutional layers and two pooling layers. The model concatenates the output of four branches and uses fully connected layers to produce a 1 × 2 vector output representing the possibilities of transients and artifacts. In the next year, Reyes et al. [14] enhanced this model by proposing a measure that assesses the rotational invariant effect on the Layerwise Relevance Propagation (LRP) relevance heatmaps.
The Recurrent Neural Networks (RNNs) [15] classify supernovae from a different perspective. RNNs can process complicated sequential data and are capable of learning about the light curves of supernovae. Charnock et al. [16] tested the performance of the Long Short-Term Memory (LSTM) [15] and the Gated Recurrent Unit (GRU) [17,18]. They have achieved better results than other algorithms. Muthukrishna et al. [7] introduced the RAPID architecture using GRUs to achieve fine-grade supernovae classification. Moller et al. [19] proposed to use RNN with the Bayesian training method. They compared their work with Support Vector Machine (SVM) [20], CNN, and non-Bayesian RNN.
All image classification based methods mentioned above encountered a problem-the candidates have to be selected and cropped in the center approximately. If the candidate is located off-center in the image, the performance of the model will decrease. Burke et al. [8] applied object detection on the astronomical source classification. Their object detection framework is based on the Mask-RCNN [21] and treat the classification task as a star/galaxy detection task and achieved performance with 92% precision and 80% recall on stars and 98% precision and 80% recall on galaxies. Because of the object detection, their model can predict the category and location of the target when images with different input sizes are input.

Object Detection
Unlike common neural networks, the object detectors have an extra head predicting class and bounding box positions in addition to the backbones. The detection frameworks are classified into one-stage detectors and two-stage detectors. One-stage detectors output the confidence of the classes and the positions directly, while two-stage detectors have to propose the possible position of the objects and then classify them. Usually, the two-stage detectors have higher accuracy but are slower than the one-stage detectors.
The representative two-stage object detection networks are R-CNN [22] based detectors. The Faster-RCNN [23] is one of the most excellent detectors for a long time. The Faster-RCNN uses an extra layer called Region Proposal Network (RPN) to provide objects for the classification network. The RPN layer provides 2000 anchors and chooses the most possible targets for classification, and is much more efficient than the Selective Search algorithm. At the end of the RPN layer, the author applies an ROI pooling layer to map the proposed targets to the feature map. Because most of the works are finished by neural networks, the detection speed and accuracy have been significantly improved. The Mask-RCNN [21], the segmentation version of the Faster-RCNN, is also used in Burke's work [8] and has great performance.
On the other side, the one-stage detectors such as You Only Look Once (YOLO) [24], RetinaNet [25], CornerNet [26], FCOS [10] and so on, have caught up with two-stage detectors recently. The main idea of YOLO algorithms is to assign the classification task to the pre-defined grids. The images are divided into S × S grids and the bounding box is determined by the grid with the highest IOU. The detecting speed of the YOLOv1 can reach 45 fps. In these YOLO architectures, YOLOv3 [27] is the most widely used detectors on a variety of devices for its speed and accuracy. The author increases the number of the backbone layer to 53 (Darknet53 [27]) and uses an independent logistic regression classifier instead of softmax. The YOLOv3 runs even faster and keeps similar performance to the SSD [28]. The YOLOv3 has a relatively better performance than YOLOv1 for detecting small objects. The FCOS [10] attempts to achieve high accuracy without pre-defined anchors, which eliminates several limitations caused by anchors. It uses the Center-ness layer to suppress the generation of these negative bounding boxes, which greatly improves the recall.

Attention Mechanism
The attention mechanism has been proposed for a long time, but the seminal work [29,30] makes this technology widely known. The attention mechanism is widely used in recurrent and convolutional neural networks. The main idea of this technique is to redistribute the weights so that the neural network can focus more on specific features. There are two types of attention mechanism, that is, hard attention and soft attention. Hard attention directly sets the weight of the region of no interest to 0, while soft attention determines the weight distribution in the range between 0 and 1 smoothly and increases the weight of the focused area. Soft attention is more suitable for gradient calculating and therefore is widely used in neural network research. In 2017, Hu et al. [31] proposed an attention module called Squeezeand-Excitation (SE) block, which enables neural networks to focus on different channels. The SE module calculates representative values for different channels and multiplies these values to the channels to redistribute weights. Their later work, the SENet, won the championship of the ImageNet. Li et al. [32] presented an attention structure that can focus on the channels generated with different kernel sizes, or feature maps with different receptive fields in other words. In addition to the channel-wise attention mechanism, Wang et al. [33] tried to deal with the information missing problem and proposed a spatial attention module. Their module is a parameter-free spatial attention layer generating positional weight and activates back to the model. Woo et al. combined the channel-based attention and the spatial attention and proposed the Convolutional Block Attention Module (CBAM) [34]. The CBAM achieves better results on the ImageNet-1k dataset. In [35], Chen et al. presented a method that integrates RetinaNet and attention mechanism to generate the video synopsis for long captured videos.
In their experiments, the combined attention method improved the Average Precision (AP) of detection by about 3%. The Center-Mask [36] is an efficient anchor-free instance segmentation algorithm based on the FCOS, which uses an attention-guided mask (SAG-Mask) in the detectors. They used effective Squeeze-and-Excitation (eSE) instead of the original SE module and achieved the highest mask AP of 38.3%.

Datasets
Most of the data used by the existing works, as mentioned in Section 2, are simulated data or partially generated by simulation. In [7,8,12,16,19], the positive candidates for training are simulated. In [13,14], the authors simulated observation fluctuations such as flux and SNR variation. Currently, there is a lack of publicly available real data for the supernovae detection research, which hinders the development of this field. In this section, we built two datasets-the PS1-SN dataset and the PSP dataset. All the images of both datasets come from real observation instead of generating by simulation. Different from the simulated data, the images in our datasets include various defects caused by the actual observation process.

PS1-SN Dataset
Our first dataset is referred to as the Pan-STARRS1 SuperNova (PS1-SN) dataset comes from the pipeline of the Panoramic Survey Telescope and Rapid Response System for Supernovae (https://star.pst.qub.ac.uk/ps1threepi/psdb/, accessed on 26 May 2020), a system for wide-field astronomical imaging.
We collect about 12,447 difference images from the known object list. We have labeled all the supernovae in these images and constructed the PS1-SN dataset. Some samples in this dataset are displayed in Figure 1. All of the images have the same size of 302 × 302 and most of the targets are in the center of the images. The images from this program have relatively high quality. They are evenly exposed and their backgrounds are free of debris. As a result, supernovae are very obvious in this dataset. Compared with the PSP-SN dataset to be introduced later, detection models trained with the PS1-SN dataset result in better performance.  •

PSP-SN Dataset
Most of the images have defects. Typical defects in this dataset are shown in Figure 2, which are caused by low SNR rate, artifacts, equatorial mounts failure, unaligned, different brightness, bad weather, and so on. While these defects increase the variety of data, they also bring about complexity to achieve high detection accuracy. • The number of samples provided by the PSP project is insufficient. Deep learning methods usually require a large number of samples to train models with high accuracy.
• The images from the PSP are cropped into different sizes. The image sizes of the PSP range from 288 × 288 to 950 × 950. In the later experiments, we will apply data augmentation to the PSP-SN dataset to improve the performance of the detection model. We augmented the original PSP-SN dataset offline with Gaussian blur, rotation, distortion, horizontal, vertical flip and so on. Some augmented samples are displayed in Figure 3. Eventually, we obtain an augmented version of the PSP-SN dataset with 12,384 images. For convenience, the original PSP-SN and the augmented PSP-SN dataset will be referred to as the PSP-Ori and the PSP-Aug later.

Methods
Our supernovae detection neural network is based on the FCOS framework enhanced by attention mechanism, which is displayed in Figure 4. In this section, we explain the basic FCOS framework, the enhance with attention mechanism and manipulation with image input size.

The FCOS Framework
The FCOS [10] is an anchor-free detector, since it predicts a 4-dimensional vector denoted as V(l, t, r, b), which contains the distances to left, top, right, bottom of each position. After the feature maps of different stages in the pyramid are extracted, they are sent to the FCOS head for further processing. The FCOS head consists of the regression branch and classification branch, both of which contain 4 convolutional layers. These convolutional layers transform the weights into classification vector and regression vector. To enhance the performance, the weights of the off-center bounding boxes are suppressed by a layer called the Center-ness. The Center-ness is a parallel branch to the classification output. The Center-ness layer uses the following formula to reduce the weight of the edge area and can greatly improve the recall of the FCOS.
where l * , r * , t * , b * denotes the predicted elements of V(l, t, r, b). The Center-ness produces a vector of the same size as the feature image. The value of the center in this vector is 1, and gradually reduces to 0 towards the edge. The vector and classification results are multiplied by the predicted elements to get the centralized classification result.

The Attention Module
The attention mechanism is usually used to increase the weights of the useful features. There are 3 kinds of attention blocks tested in our paper: the channel-based attention (Squeeze and Excitation, SE) [31], spatial-based attention, and the combination of these two attention blocks (Convolutional Block Attention Module, CBAM) [34]. These modules are displayed in Figure 5. The channel-based attention gives different weights to different channels. The channelbased attention formula is listed below: SE(F in ) = σ(FC(δ(FC(AvgPooling(F in ))))). ( Here, we have F in ∈ R H×W×C as the input for the ith layer. First, a global average pooling (AvgPooling) operation is applied to generate a squeezed 1 × C vector and each element of the squeezed vector represents a channel-wise value. Then, a fully connected (FC) layer is used to learn the nonlinear combination characteristics. The SE module uses the ReLU function denoted as δ to activate these representative channel-wise values and the range of output values is limited to [0, 1] with a sigmoid function denoted as σ. Squeezed and excited vector can inform the neural network which channels should be focused on. In this paper, we use channel attention (CA) with both average pooling and maximum pooling (MaxPooling) as follows: CA(F in ) = σ FC(δ(FC(AvgPooling(F in )))) + FC(δ(FC(MaxPooling(F in ))))).
Eventually, the ith layer feature map is multiplied with the vector to obtain the output weighted features: The second attention we tested in our architecture is the spatial attention. The main idea of spatial attention is to generate a weighted feature map to represent the importance of different positions crossing channels. The summarized spatial attention is listed below: where notation "|" is the operation of concatenation, and Conv 7×7 denotes a convolutional layer with a kernel size of 7 × 7. As in [34], an average pooling and maximum pooling are used to combine all the channels, then the output feature map is followed by a convolutional layer with a kernel size of 7 × 7 to calculate positional feature map. The final output is then: The last attention is the combination of the two methods mentioned above, that is, CBAM. The CBAM uses the SE module first to generate channel attention, and then uses the spatial attention mechanism to generate positional attention. The shortcut links the input layer directly to the output layer:

Increase Input Size
According to our analysis, most of the targets in our datasets are small objects (smaller than 32 × 32 pixels). Therefore, small object detection techniques are applied in our architecture. The size of the feature map becomes smaller because of the convolution operation and this phenomenon is more serious in deep convolution networks. The direct result of this phenomenon is that small targets lose most of the details at the end of deep convolutional neural networks. Therefore, the performance of the detector for small object detection is usually the bottleneck of the detection task. One of the basic techniques is to increase the sizes of input images. We initially set the image input size as 416 × 416, then increase to 800 × 800, and finally to 960 × 960.

Experiment
In this section, we first compare different object detection algorithms on the datasets and select the best one as the baseline. Then we improve the selected algorithm on the PSP dataset with some of the small object detection techniques.

Model Selection
We compare Deep-Hits (DH) [12], Cycling Average Pooling (CAP) [14], YOLOv3 and FCOS during model selection, as shown in Table 1. DH and CAP are designed specifically for supernovae detection, while YOLOv3 and FCOS are used for general object detection. First, we evaluate the DH model and the CAP model on the PS1-SN dataset. The original input size of the two rotation invariant models is 21 × 21, so we crop images in the PS1-SN dataset at the center. Then we generate negative samples by selecting the background, which produces about 12,000 negative samples. The test result shows that both of the models have high detection performance on the PS1-SN dataset. For the object detection model, we test YOLOv3 and FCOS on both the PS1-SN dataset and the PSP-Ori dataset. Both methods are tested with an original input size, which is much larger than 21 × 21. We fine-tune the training parameters for two models separately to achieve their best performance, as shown in Table 2. For the PS1-SN dataset, the two models perform similarly, where YOLOv3 has slightly higher precision and FCOS has a slightly higher recall. Both methods outperform the DH and CAP model. For the PSP-Ori dataset, all metrics are much lower. Since F1 scores of FCOS are higher than YOLOv3, we focus on the FCOS as our baseline.

Performance Enhancement
We use several improvement techniques to achieve a better performance on this task. According to the previous results, there is little improvement space for PS1-SN, where the performance on PSP-Ori is not good enough. Therefore, we use PSP datasets for evaluation in this part. As shown in Table 3, the data augmentation improves the performance a lot. To make the neural network focus on the feature of interest, we apply the attention mechanism to the FCOS. The training parameters are selected as follows-the batch size is all set to 8, the weight decay is 1 × 10 −4 , the learning rate is initialized to 1 × 10 −3 and is reduced by a factor of 10 at the iteration of 14,000 and 20,000. We trained all methods in 25,000 iterations. We have tested three kinds of attention mechanisms, that is, the channel attention, the spatial attention, and the CBAM, with different input sizes. When the input size is 416 × 416, all attention mechanisms produce a negative effect on the performance. We then try to increase the size of the input image, which begins to work out. At first, the performance improvement is not obvious when the maximum image size is 800 × 800. When the maximum size is further increased to 960 × 960, the performance is enhanced. The spatial attention performs best this time, while the other methods still have the same or negative effect. We also investigate the effect of the above enhancements on the YOLOv3, by conducting similar experiments of YOLOv3 as FCOS. As shown in Table 4, the detection performance of YOLOv3 improves with the size of the input image, but the improvement is not obvious when the input size changes from 800 to 960. The CBAM is counterproductive in the results, and the effects of spatial and channel attention are not obvious. Comparing Tables 3 and 4, we can conclude that the enhanced FCOS achieves a better performance on the PSP-SN dataset. The network visualization heat map displays the effects of channel attention and spatial attention. The data from the heat map comes from the output right after the attention module or residual blocks. We extract the feature map from the 1st, 5th, and 11th residual block of the enhanced FCOS. As Figure 6 shows, the attention module successfully attaches more weights to the target than the original network. Specifically, the advanced networks are more interested in regions of candidate supernovae.

Discussion
We list some methods that we tried but that did not work here. These methods include some neural network module designs and hybrid dataset techniques. Though these methods do not have a positive influence on the supernovae detection task, they still provide some ideas for solving relevant problems.

•
Upsampling: As described in Section 3, supernovae detection suffers from feature loss in deep convolutional neural networks. We tried upsampling for the feature map with a ratio of 1.2 at the output of each feature pyramid stage and reduced the size shrinkage of the feature map to some extent. However, such a method did not improve much performance. • Hybrid dataset: This idea is inspired by the article [9]. The author argues that training data for the relevant research lacks statistical representation. To figure out whether the practical representative samples can enhance the performance or not, we blend the two datasets of PSP-Aug and PS1-SN as a hybrid dataset. We test the performance of FCOS with two blending ratios of PSP-Aug to PS1-SN, which are 1:20 and 1:1. The models trained with the hybrid dataset are also fine-tuned to achieve the best performance and the results are listed in Table 5. When the blending ratio is increased, the performance with the PSP-Aug decreases severely while the performance with the PS1-SN does not change too much. This result shows that the detection architecture tends to have good performance on the data with low complexity, while causes performance degradation on the complex data. Another problem to be discussed is the ability of the model to detect different types of defect images. We save the detection results of the testing set for analysis. Figure 7 shows several examples of false positive results. The green bounding-boxes mark the targets, and the category and confidence are labeled beside the bounding-boxes. Although the model can discover the features of unaligned stars to some extent, as shown in Figure 7a, the model produces most of the false positive results on the unaligned images, as shown in Figure 7b,c. At the same time, dim target and noise cause miss detection, which is the main reason for a low recall rate. Figure 8 displays some cases of dim supernovae, and the target is marked by a red arrow. The above results show that the model has defects in the detection of unaligned and dim targets. Therefore we need to pay more attention to alignment and exposure in the data pre-processing stage of the supernovae detection system.

Conclusions
In this paper, we studied the problem of supernovae detection, and compare two potential object detection frameworks on this task. We built two new supernovae detection datasets, which consist of 12,447 images and 716 images respectively. We believe such datasets will benefit the relevant research community. The FCOS achieves the performance of 99.6% precision and 98.6% recall on the PS1-SN dataset and 84.9% precision and 51.4% recall on the PSP dataset. Experimental results show that the FCOS performs better than YOLOv3 on both datasets, so the FCOS selected as the baseline is used to further improve the performance for supernovae detection according to the feature of the supernova. In our experiment, the original PSP dataset is augmented to 12,384 images with a series of techniques and generates obvious improvement on our task. Here we have tested different attention blocks and studied the effect of different input image sizes on the detecting performance. Eventually, the FCOS with spatial attention module and bigger size of input reaches 96.2% precision and 64.3% recall, which is the best performance we can get in our work. Besides, we have tried some other ideas, but they do not have the effect of improving performance. By using the object detection framework, we can implement the automatic processing of supernovae detection tasks and thus accelerate the processing pipeline of observation data.
Object detection framework is a flexible object extractor and classifier. Especially in the complex scene, the automatic extraction and detection of objects are particularly important. Compared with the traditional neural network with only a few layers, the object detection framework can achieve high-precision and real-time detection based on deep convolutional networks to deal with the information represented by complex circumstances. We suggest that future research should be based on a deeper convolution network and adopt small target detection technology. One limitation of our research is that we do not have access to multi-channel images for these targets. Multi-channel images such as UVBIR images are often used in astronomical images. These images contain infrared and ultraviolet channels that may provide more information about supernovae. Therefore, in our future research, we will seek opportunities to obtain multi-channel images from real scenes to enhance our datasets. Besides, we found image size has an impact on the detection performance of FCOS. We will investigate how to automatically and efficiently select the best image size in the future.


extfieldsofstudy:
Computer Science
Mathematics

text:
Concatenated Codes for Recovery From Multiple Reads of DNA Sequences

Decoding sequences that stem from multiple transmissions of a codeword over an insertion, deletion, and substitution channel is a critical component of efficient deoxyribonucleic acid (DNA) data storage systems. In this paper, we consider a concatenated coding scheme with an outer low-density parity-check code and either an inner convolutional code or a block code. We propose two new decoding algorithms for inference from multiple received sequences, both combining the inner code and channel to a joint hidden Markov model to infer symbolwise a posteriori probabilities (APPs). The first decoder computes the exact APPs by jointly decoding the received sequences, whereas the second decoder approximates the APPs by combining the results of separately decoded received sequences. Using the proposed algorithms, we evaluate the performance of decoding multiple received sequences by means of achievable information rates and Monte-Carlo simulations. We show significant performance gains compared to a single received sequence.

I. INTRODUCTION
Coding-theoretic aspects of data storage in deoxyribonucleic acid (DNA) have recently gained significant attention. This new line of research has been triggered by a manifold of successful experiments [1]- [3] that demonstrated the viability of DNA as a reliable medium for archival data storage. Following the pioneering experiments, a variety of related information-theoretic problems with different foci have been identified. One of these problems is the reliable transmission over channels that introduce insertions, deletions, and substitutions (IDSs) [4], motivated by the fact that the sequencing process often introduces errors in the form of IDSs.
There is a variety of literature dedicated to the study of error-correcting codes for the IDS channel. While the list of related papers is long, the line of research most relevant to our work is the one inspired by Davey and MacKay [5]. In their paper, they propose a class of concatenated codes together with a decoding algorithm that represents the inner code and the channel by a hidden Markov model (HMM), which allows inferring a posteriori probabilities that are passed to the outer decoder. Following this work, the Markov process of convolutional codes was incorporated into that of the IDS channel, allowing to run standard decoding algorithms of convolutional codes for the IDS channel [6], [7]. The approach of [5] was later refined with an improved decoding algorithm [8]  using inner marker codes [9]. Recently, it was shown that synchronization is also possible without an inner code using a standalone low-density parity-check (LDPC) code [10].
The vast majority of previous studies has focused on the case of error correction after a single transmission over an IDS channel. In DNA-based storage, however, data is typically synthesized into many short DNA strands, and each of these strands is replicated thousands of times. Therefore, when accessing the stored data via DNA sequencing, multiple copies of each original strand are read. With this in mind, in this paper we deviate from previous work and consider error-correcting codes for multiple transmissions of a coded sequence over a channel that is impaired by IDS errors. In particular, we propose two new decoding algorithms for multiple received sequences and show that significant gains can be achieved as compared to a single transmission. The first decoding algorithm, mainly a benchmark for the second decoder, is an optimal symbolwise a posteriori probability (APP) decoder. While this algorithm has exponential complexity in the number of received sequences, we show that it is possible to achieve competitive performance with a second, practical decoder that has only linear complexity. For our results we employ a new concatenated coding scheme with an outer LDPC code and an inner convolutional code and compare with the watermark codes from [5]. To the best of our knowledge, this is the first work to propose decoding algorithms for the case of coded multiple transmission and to evaluate coding schemes as well as achievable information rates (AIRs) over this channel.
Related to our work is the study of sequence reconstruction, introduced by Levenshtein [11]. Here, a sequence is repeatedly transmitted over an erroneous adversarial channel and the goal is to analytically quantify, as a function of the sequence length, the number of sequences that are required to guarantee correct reconstruction of the transmitted sequence with zero error probability. Levenshtein's initial work on insertion and deletion errors has recently been extended by additionally allowing substitution errors [12].
Similar in spirit to our work, yet different in methods and objectives, is the research of trace reconstruction over deletion channels [13], [14], which is the probabilistic variant of the sequence reconstruction problem. While these works mainly derive asymptotic results, here we discuss a fixed number of short, finite-length received sequences over a channel that additionally allows insertions and substitutions. More recently, the trace reconstruction problem has also been formulated Delete: y ← y

Transmit
No error: for a fixed number of sequences with a larger focus on algorithmic aspects [15], [16]. However, these works consider only uncoded sequences, while we are interested in the case of coded transmission. Finally, in [17] Varshamov-Tenegolts codes have been proposed for error correction over multiple channels that introduce a fixed number of deletions.

A. Channel Model
We consider a state-based representation of the IDS process [5], [18] as depicted in Fig. 1. Let x = (x 1 , . . . , x N ), x i ∈ Σ q = {0, 1, . . . , q − 1} be the sequence to be transmitted over the channel. The transmitted symbols x i are successively queued for transmission and the received sequence y is assembled by the following procedure. When x i enters the queue, the channel enters state x i and there are three different possible events. First, a uniformly random symbol a ∈ Σ q is inserted to the received sequence y with probability p I . In this case, x i remains in the queue and the channel returns to state x i . Second, the symbol x i is deleted with probability p D without appending it to y and the next symbol x i+1 is enqueued. Third, x i is transmitted with probability p T = 1 − p I − p D . In this case, the symbol is substituted with a random symbol a * = x i with probability p S and the next transmit symbol x i+1 is enqueued. When the last transmit symbol x N leaves the queue, the procedure finishes and the channel outputs y. Note that y = (y 1 , . . . , y N ) has random length N , depending on the channel realization.

B. Forward Error-Correction Scheme
In this work, we will use a concatenation of an outer code with an inner code as our forward error-correction scheme, following [5]. The role of the inner code is to maintain synchronization and provide reliable soft information to the outer code, while the role of the outer code is to use this soft information to perform an accurate estimate of the transmitted information. In particular, given that the inner code provides reliable synchronization information, the outer code corrects the remaining errors on the synchronized sequence. In our scheme, the inner code is either a convolutional code as used in [19], or a watermark code as used in [5]. Note that the watermark code can be thought of as a convolutional code with memory equal to zero, and we will therefore restrict some passages to convolutional inner codes only. The outer code is either a binary or a nonbinary LDPC code.
The resulting codeword w = (w 1 , . . . , w No ), w i ∈ F 2 k is then encoded via either an [n, k, m] q inner convolutional code of blocklength n, binary input length k, memory m, and output alphabet Σ q or an [n, k] q inner block code. With this concatenation, each outer codeword symbol w i ∈ F 2 k is mapped to n symbols over Σ q . We terminate the convolutional code, resulting in the codeword The inner code consequently has rate R I = Nok N log q , and thus we transmit at an overall rate R = R o R I = Kk N log q . Before entering the IDS channel, the codeword v is offset by a pseudo-random sequence resulting in the final output x = (x 1 , . . . , x N ), x i ∈ Σ q . This offset sequence is known to the inner decoder, and is used to maintain synchronization at high probabilities of insertions or deletions. According to the nomenclature in [5] we call the combination of the inner block code and offset watermark code.
The output sequence x is then transmitted M times independently over an IDS channel resulting in received sequences y 1 , . . . , y M corresponding to M reads of the original strand. The inner decoder uses these received sequences to infer likelihoods for the symbols in w. These likelihoods are then fed to the outer decoder, which decides on the decoded sequenceû. Furthermore, we can also iterate between the inner and outer decoder, exchanging extrinsic information between them. The communication system is depicted in Fig. 2.

III. SYMBOLWISE A POSTERIORI DECODING OF INSERTION/DELETION/SUBSTITUTION CHANNELS
One of the main challenges for decoding from IDS channels is the random memory associated with the insertion and deletion process. This is visualized by the receiver's inability to directly identify the origin of a received symbol y i . Since insertions or deletions before symbol y i might have moved the symbol right or left in the received sequence, y i could be the result of transmitting a symbol x i with i = i. In the following, we describe how it is still possible to infer a posteriori likelihoods from this channel using a hidden Markov representation of the channel [5], [7], [8]. We present and discuss this HMM for M = 1 first and extend it to M > 1 multiple received sequences afterward. Here we restrict the discussion to convolutional inner codes, as the watermark code can be viewed as a convolutional code with m = 0.

A. Single Received Sequence
The APPs of the outer codeword symbols are given by The standard approach to compute p(y, w i ) for convolutionally encoded transmit sequences is to de-marginalize the distribution p(y, w i ) with respect to the memory states of the convolutional code that correspond to the information symbol w i and then exploit the Markov property of the convolutional code. That is, given the memory state at time i, the output after time i becomes independent of earlier states and the preceding output symbols. However, for the case of the IDS channel, this Markov property is violated through the memory of the channel. Therefore, it is useful to add a new hidden state variable, the so-called drift [5]. The key property of the drift is that, by its inclusion as a new state variable inside the Markov process, the Markov property is restored. This is because the drift sequence itself forms a Markov chain and, conditioned on the memory state and drift at time i, the output of the channel after time i becomes independent of previous memory states and drifts. Denoting the state variables of the convolutional code by s i , i = 0, 1, . . . , (N o +m), and introducing the joint state variable σ i = (s i , d i ), we obtain by slight abuse of notation p(y, w i ) = (σ,σ ):wi p(y, σ, σ ) , where the summation is over all state transition pairs (σ, σ ) that correspond to the information symbol w i . Further, due to the Markov property, we can expand the joint probability to p(y,σ,σ ) = p y (i−1)n+d 1 , σ p y in+d (i−1)n+d+1 , σ σ p y N in+d +1 σ . Abbreviating the above terms by α i−1 (σ), γ i (σ, σ ), and β i (σ ) in order of appearance, one can deduce the forward and backward recursions with appropriate termination conditions incorporating the termination of the convolutional code and drift states. The branch metric γ i (σ, σ ) = p(w i )p(y in+d (i−1)n+d+1 , d d, s, s ) can be efficiently computed using a lattice structure, see, e.g., [7], [20] 1 using the transmit sequence corresponding to the state transition s → s . Having these expressions in hand it is possible to compute the APPs p(w i |y) using the BCJR algorithm [21]. Note that while the introduction of the drift state variable allows to exactly compute the APPs, it comes at the cost of additional complexity in terms of a larger number of states and state transitions, as discussed in Section III-C.

B. Multiple Received Sequences
In this work we consider the following two approaches of computing the a posteriori distribution p(w i |y 1 , . . . , y M ) for multiple sequences.
Joint decoding: Similar to the approach for a single received sequence, it is possible to introduce a drift state sequence The forward, backward recursion, and the terminal conditions are according to the single sequence case.
Separate decoding: Instead of computing the a posteriori likelihoods precisely, we now propose a different approach that allows to approximate the joint a posteriori likelihood using the marginal likelihoods p(w i |y j ) as The approximation is motivated by the case of memoryless channels, where the proportionality holds exactly. The individual APPs p(w i |y j ) can be computed as in the case of a single received sequence by means of a BCJR decoder. While the approximation comes at reduced complexity as compared to the joint decoding approach, see Section III-C, it also results in a loss of performance, which we will quantify in Section IV. Note that it is also possible to separate the joint a posteriori likelihoods at other stages, such as directly between the channel and inner code or after decoding the LDPC code. However, we observed that for the considered parameters the proposed stage yields the best results.

C. Complexity Considerations
The number of operations performed by the inner BCJR decoder is proportional to the total number of edges of the trellis that is inferred from the HMM. In order to reduce its computational complexity, the drift states are usually limited to a fixed interval d i ∈ [d min , d max ] and the maximum number of insertions per symbol is limited to I max [5]. Denote by ∆ = d max − d min + 1 the total number of drift states and by δ = n(I max + 1) + 1 the number of possible drift transitions. The complexity, measured by the total number of edges in the trellis, of joint decoding is then given by The [1, 1, 2] 4 convolutional code is a binary convolutional code with generator polynomials (1 + D 2 , 1 + D + D 2 ), where each two bits are combined to one DNA symbol. The [3,3] 4 watermark code is constructed as in [5].
while the complexity of separate decoding is simply given by M times the complexity of decoding a single sequence, Note that here we neglect the effect of trellis termination on the number of edges. Typical values of ∆δ for the DNA storage application can easily be as high as ∆δ ≈ 1000. This means that decoding the inner code is the main contributor to the decoding complexity and we thus neglect the complexity of the outer decoder in the following. It further becomes evident that the joint decoding approach only works for a very small number of sequences M due to its exponential complexity.
IV. SIMULATION RESULTS In our simulations, we compare the two following set-ups. First, an inner convolutional code with an outer WiMAX binary LDPC code (CC + B-LDPC) and, second, an inner watermark code with an outer WiMAX-like nonbinary LDPC code (WM + NB-LDPC). Their performance is compared over a single and multiple IDS channels. Recall that in both schemes a pseudo-random offset sequence is used to improve the synchronization capability.

A. Simulation Parameters
Our simulations are done over an alphabet of size q = 4, which equates to the four bases {A, C, G, T} used in DNA and we set p S = 0 and p I = p D throughout. The code parameters are summarized in Table I. The outer LDPC code is decoded via belief propagation with a maximum number of 100 iterations. Regarding the inner code, we set I max = 2 and the limit of the drift random variable in decoding is set to be five times the standard deviation of the final drift at position N , i.e., d max = −d min = 5 √ N p D 1−p D . For multiple transmissions, we use M = 2, 5, 10, and 20. Fig. 3 shows the frame error rate (FER) of the CC + B-LDPC scheme for single and multiple transmission over an IDS channel, while Fig. 4 shows that of the WM + NB-LDPC scheme. Interestingly, when transmitting over a single IDS channel, the CC + B-LDPC scheme outperforms the WM + NB-LDPC scheme. However, for transmission over multiple IDS channels, this behavior reverses, and the WM + NB-LDPC scheme has superior performance for a larger number of sequences. While this result seems surprising at the first glance, there is a natural explanation for this phenomenon as follows. First notice that with the inner convolutional code, bitlevel APPs p(w i |y j ) with w i ∈ F 2 are computed, while the watermark code works over nonbinary inputs w i ∈ F 2 3 . Now, for larger error probabilities p I , p D , the memory of the channel  becomes more apparent and thus also the a posteriori distribution has increased correlation between different input symbols w i . While this correlation is marginalized for the binary APPs, it is better captured by the nonbinary input symbols and thus the WM + NB-LDPC scheme achieves better performance for larger number of sequences or higher error probabilities. In terms of complexity, both schemes have the same ∆ and M , but different n, k, m, and δ. Comparing both complexities for equal blocklengths, we get C conv.

B. Discussion of Numerical Results
sep /C watermark sep ≈ 1.2. Thus, with these parameters the convolutional code is slightly more complex than the watermark code. Since the complexity increases exponentially with the memory m, larger memory convolutional codes are expected to become even more complex as compared to the watermark code. The same conclusion can be made when multiple channel transmissions are combined with inner-outer iterative decoding, since doing M iterations has the same complexity as separate decoding of M received sequences. Fig. 5 shows the FER of both schemes when iterating times between the inner and outer decoder. We fix the total complexity such that the product of the number of iterations and the number of sequences is equal to 20. It is observed that the iterative scheme benefits more from a larger number of received sequences, however in practice this comes at the cost of additional sequencing costs. Finally, AIRs with an inner convolutional code are displayed in Fig. 6. In agreement with our previous results, the achievable rates are significantly larger for multiple transmissions as compared to a single transmission. Not surprisingly, the joint decoder achieves higher rates. It exhibits a similar performance with M = 2 transmissions as compared to the separate decoder with M = 5 sequences.

V. FUTURE WORK
In view of the huge number of trellis states that the joint decoding approach entails, an immediate idea is to consider suboptimal decoding of the inner code, such as sequential  decoding. There are however some obstacles that have to be overcome, including a huge number of edges per node and an appropriate metric. This will be addressed in future work.


FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation

Recent methods for long-tailed instance segmentation still struggle on rare object classes with few training data. We propose a simple yet effective method, Feature Augmentation and Sampling Adaptation (FASA), that addresses the data scarcity issue by augmenting the feature space especially for rare classes. Both the Feature Augmentation (FA) and feature sampling components are adaptive to the actual training status -- FA is informed by the feature mean and variance of observed real samples from past iterations, and we sample the generated virtual features in a loss-adapted manner to avoid over-fitting. FASA does not require any elaborate loss design, and removes the need for inter-class transfer learning that often involves large cost and manually-defined head/tail class groups. We show FASA is a fast, generic method that can be easily plugged into standard or long-tailed segmentation frameworks, with consistent performance gains and little added cost. FASA is also applicable to other tasks like long-tailed classification with state-of-the-art performance.

Introduction
A growing number of methods are proposed to learn from long-tailed data in vision tasks like face recognition [19], image classification [31] and instance segmentation [15]. We focus on the problem of long-tailed instance segmentation, which is particularly challenging due to the class imbalance issue. State-of-the-art methods [2,5,16,27] often fail to handle this issue and witness a large performance drop on rare object classes. Fig. 1 exemplifies the struggle of the competitive Mask R-CNN [16] baseline on highly imbalanced LVIS v1.0 dataset [15]. We can see that there are about 300 rare classes in the tail that have no more than 10 2 positive object instances. Such scarce training data result in poor performance for most of the tail classes, with 1 GitHub: https://github.com/yuhangzang/FASA. 2 Project page: https://www.mmlab-ntu.com/project/fasa/index.html. near-zero class probabilities predicted for them.
To address the data scarcity issue, one intuitive option is to over-sample images that contain the tail-class objects [15]. But the downside is that the over-sampled images will include more head-class objects at the same time due to the class co-occurrence within images. Hence for the instance segmentation task, re-sampling at the instance level is more desirable than at image level. Another option is data augmentation for considered objects, either in image space (e.g., random flipping) or in feature space (i.e., feature augmentation, on object regional features). Along this line, many methods have proven effective by augmenting the image/feature space of rare classes. The feature augmentation methods already show benefits for face recognition [48,28], person Re-ID [28] and classification [22,8]. However, these methods require manual design of class groups, e.g., using heuristics like class size. And they often involve two stages of feature training and feature transfer, which induce additional cost.
Feature augmentation remains little explored for longtailed instance segmentation tasks. In this paper, we present an efficient and effective method, called Feature Augmentation and Sampling Adaptation (FASA). FASA does not require any elaborate transfer learning or loss design [25,37,45]. As a result, FASA keeps its simplicity, has extremely low complexity, while remains highly adaptive during training. In the proposed method, we perform online Feature Augmentation (FA) for each class. The FA module generates class-wise virtual features using a distributional prior with its statistics calculated from previously observed real samples. This allows FA to capture class distributions and adapts to evolving feature space.
The augmented features can still be limiting for rare classes. This is because the observed feature variance is often small with few training examples. To overcome this problem, we adapt the sampling density of virtual features for each class. This way, our synthesized virtual features still live in the true feature manifold, just with different sampling probabilities to avoid under-fitting or over-fitting. We propose an adaptive sampling method here: when aug- mented features improve the corresponding class performance in validation loss, the feature sampling probabilities get increased, otherwise decreased. Such a loss-optimized sampling method is effective to re-balance the model predicting performance, see Fig 1(b). It is noteworthy that our full FASA approach handles virtual features only, thus it can serve as a plug-and-play module applicable to existing methods that learn real data with any re-sampling schemes or loss functions (standard or tailored for class imbalance). Comprehensive experiments of instance segmentation on LVIS v1.0 [15] and COCO-LT [42] datasets demonstrate FASA as a generic component that can provide consistent improvement to other methods. Take LVIS dataset as an example. FASA improves Mask R-CNN [16] by 9.0% and 3.3% in mask AP metrics for rare and overall classes respectively, and improves a contemporary loss design [37] by 10.3% and 2.3%. Moreover, these gains come at the cost of merely a ∼ 3% increase in training time, see Fig. 1(c). Furthermore, FASA can generalize beyond the instance segmentation task, achieving state-ofthe-art performance on long-tailed image classification as well.
To summarize, the main contributions of this work is a fast and effective feature augmentation and sampling approach for long-tailed instance segmentation. The proposed FASA can be combined with existing methods in a plug-and-play manner. It achieves consistent gains and surpasses more sophisticatedly designed state-of-the-art methods. FASA also generalizes well to other long-tailed tasks.

Related Work
Long-tailed classification. For the long-tailed classification task, there is a rich body of widely used methods including data re-sampling [4] and re-weighting [3,9]. Recent works [21,52] reveal the effectiveness of using different sampling schemes in decoupled training stages.
Instance-balanced sampling is found useful for the first feature learning stage, which is followed by a classifier finetuning stage with class-balanced sampling. Long-tailed instance segmentation. To handle class imbalance in instance segmentation task, recent methods still heavily rely on the ideas of data re-sampling [15,34,18,45], re-weighting [37,33,36,41,54,43,51] and decoupled training [25,42]. For re-sampling, class-balanced sampling [34] and Repeat Factor Sampling (RFS) [15] are conducted at the image level. However, image-level re-sampling sometimes worsens the imbalance at instance level due to the instance co-occurrence within images. The methods of data-balanced replay [18] and NMS resampling [45] belong to the category of instance-level resampling. For the re-weighting scheme, Equalization loss v1 [37] and v2 [36] are such representative methods that re-weigh the sigmoid loss. More recent works [25,45] attempt to divide imbalanced classes into relatively balanced class groups for robust learning. However, the class grouping process relies on static heuristics like class size or semantics which is not optimal. Tang et al. [38] studied the sample co-existence effect in the long-tailed setting and proposed de-confounded training. Seesaw Loss [41] dynamically rebalances the gradients of positive and negative samples, especially for rare categories. Surprisingly, data augmentation as a simple technique, has been barely studied for long-tailed instance segmentation. In this paper, we show that competitive results can be obtained from smart data augmentation and re-sampling in the feature space, while being intuitively simple and computationally efficient. Also our approach is orthogonal to prior works, and can be easily combined with them to achieve consistent improvements. Data augmentation. To avoid over-fitting and improve generalization, data augmentation is often used during network training. In the context of long-tailed recognition, data augmentation can also be used to supplement effective training data of under-represented rare classes, which helps to re-balance the performance across imbalanced classes.
There are two main categories of data augmentation methods: augmentation in the image space and feature space (i.e., feature augmentation). Commonly used imagelevel augmentation methods include random image flipping, scaling, rotating and cropping. A number of advanced methods have also been proposed, such as Mixup [50] and Cutmix [49]. For our considered instance segmentation task, the copy-and-paste technique such as InstaBoost [13] and Ghiasi et al. [14] and prove effective. More recent works try to synthesize novel images using GAN [30] or semi-supervised learning method [46]. Compared to the well established image-level data augmentation technique in deep learning, feature augmentation has not yet received enough attention. On the other hand, Feature Augmentation (FA) directly manipulates the feature space, hence it can reshape the decision boundary for rare classes. Classic FA is built on SMOTE [4]-typed methods that interpolate neighboring feature points. More recently, researchers propose to Manifold Mixup [39] or MoEx [23] for better performance. ISDA [44] augments data samples by translating CNN features along the semantically meaningful directions. Such methods are often not directly applied for imbalanced class discrimination and suffer from high complexity.
Some recent works have shown that feature augmentation benefits long-tail tasks such as face recognition [48], person Re-ID [28], or long-tail classification [8,22,29,24]. However, we observe that these methods have limitations when apply them to the long-tailed instance segmentation datasets such as LVIS [15]. Due to the high computational cost of instance segmentation task, some methods [48,22,8] rely on two-phase pipeline or large memory of historical features [29] incur high cost in time and memory thus less efficient. The special background class exists for the instance segmentation task (no class anchor), makes method [28] that rely on margin-based classification loss less effective. In addition, the small batch size of the instance segmentation frameworks limits the performance of approach [8] that rely on mining confusing categories. Our approach is specially designed for long-tailed instance segmentation, waives the need for complex two-phase training and the associated costs. We compare with them in our experiments, see Sec 4.1.

Methodology
In this section, we introduce the proposed Feature Augmentation and Sampling Adaptation (FASA) approach consisting of two components: 1) adaptive Feature Augmentation (FA) that generates virtual features to augment the feature space of all classes (especially for rare classes), 2) adaptive Feature Sampling (FS) that dynamically adjusts the sampling probability of virtual features for each class.
To better illustrate how FASA works for long-tailed in-stance segmentation, we take the Mask R-CNN [16] framework as our baseline and show an example of combining FASA and the segmentation baseline. The overall pipeline is shown in Fig. 2. Note FASA is a standalone module for feature augmentation, leaving the baseline module unaltered. Therefore, FASA serves as a plug-and-play module and it can combine with and facilitate stronger baselines other than Mask R-CNN. We will show this flexibility in our experiments. Under the Mask R-CNN framework, the standard multitask loss is defined for each Region of Interest (RoI): For simplicity, we apply FASA to the classification branch only, which is the most vulnerable branch on long-tailed data as supported in Fig. 1(b). More discussion is provided in the supplementary materials. It is possible to augment other branches using FASA as well. We leave that to future work and expect further improvements.

Adaptive Feature Augmentation
Mask R-CNN is originally trained on "real" feature embeddings of positive region proposals generated by the Region Proposal Network (RPN). For each class, we aim to augment its real features that are likely scarce (e.g., for rare classes). An ideal FA component should have the following properties: 1) generate diverse virtual features to enrich the feature space of corresponding class, 2) the generated virtual features capture intra-class variations accurately and do not deviate much from the true manifold which hinders learning, 3) adapt to the actual feature distribution that is evolving during training, 4) high efficiency.
To this end, we maintain an online Gaussian prior from previously observed real features, which fulfills the abovementioned requirements. We found such prior suffices for FA purpose by generating up-to-date and diverse features even when the actual features are not Gaussian. Concretely, for each foreground class c within the current batch, we can calculate the corresponding feature mean µ t c ∈ R d and standard deviation σ t c ∈ R d which together define a Gaussian feature distribution. Given the noisy nature of µ t c and σ t c , we use them to continuously update the more robust estimates µ c and σ c instead, via a momentum mechanism: where m is set to 0.1 in all experiments. Then according to a Gaussian prior with up-to-date µ c and σ c , we generate the class-wise virtual featuresx c by random perturbation under the feature independence assumption:  The feature independence assumption allows for efficient FA, where augmented features are treated as IID random variables. We consider the covariance matrix of the Gaussian to be diagonal, thus largely reduce the complexity from d 2 to d. Such generated virtual features {x c } will need to be re-sampled (detailed later). Finally, both re-sampled {x c } and real features {x c } (with its own sampling strategy in any baseline method) are sent to compute L cls in Eq. (1).

Adaptive Feature Sampling
For a wise use of the generated virtual features for each class, we propose an adaptive Feature Sampling (FS) scheme to effectively avoid under-fitting or over-fitting from FA. The sampling process operates in a relative fashion: if virtual features improve the corresponding class performance, their sampling probabilities get increased, otherwise decreased. Such relative adjustments of virtual feature sampling probabilities cater to the changing needs of FA during training. We can imagine that FA may be useful for rare classes during some training periods, but other training stages may call for decreased amount of FA to avoid overfitting. By contrast, a static predefined sampling distribution would be independent of training dynamics and thus suboptimal. Parametric sampling formulation. Note we still need an initial feature sampling distribution so as to adjust it later on. Obviously, we can benefit from a good initialization that avoids initial biased FA and costly adjustments. We choose to initialize the sampling distribution simply based on the inverse class frequency. It favors rare classes with higher sampling probabilities p c and is useful in our classimbalanced setting with minimal assumption about data distribution. Now we have a predefined skewed sampling distribution at the beginning. Then we dynamically scale the per-class sampling probability p c as follows: where s c is the scaling factor to be estimated at each time, and N c denotes the class size of class c. Adaptive sampling scheme. Recall the insight behind adapting the sampling probability p c is that: if FA does improve the performance of class c, then we should generate more virtual features and increase p c ; if we observe worse performance from FA, then p c is probably big enough and we should decrease it to avoid over-fitting with the augmented features. In practice, we use multiplicative adjustments to update p c every epoch. Specifically, we increase p c to min(1, p c · α) with α = 1.1, and decrease p c to max(0, p c · β) with β = 0.9. One can adopt different performance metrics to guide the adjustment of p c . Existing adaptive learning systems are either based on the surrogate loss [35] or the more ideal actual evaluation metric [20]. However for large-scale instance segmentation tasks, frequent evaluation of metrics like mAP on large datasets is very expensive. Therefore, we use the classification loss L cls on validation set to adjust p c for our large-scale segmentation task. For the validation set, we apply Repeat Factor Sampling (RFS) [15] method to balance the class distribution and provide more meaningful losses. The aforementioned setting works reasonably well in our experiments. Group-wise adaptation. To make our method more robust to different scenarios, we need to address two common challenges in the long-tailed instance segmentation task. First, the per-class loss can be extremely noisy on limited evaluation data, e.g., for rare classes. In this case, both loss and metric cannot serve as reliable performance indicators. Second, the evaluation data may simply not be available for all classes. For example, the validation set of LVIS dataset [15] only has 871 classes out of 1203 from training set. This makes it impossible to evaluate loss for the other 332 classes on validation set. To solve the aforementioned issues, we propose to cluster all training classes into super-groups. Then we compute the average validation loss of within-group classes, and adjust their feature sampling probabilities together. In other words, we adjust the sampling probabilities by a single scaling factor (α or β) depending on the average of group-wise loss. By doing so, those classes with missing evaluation data can be safely ignored when computing the loss average, but their sampling probabilities can still be updated along with other classes within the same group. Moreover, the group-wise update is less noisy since it is based on the loss average computed on bigger data (from multiple classes).
For clustering, rather than using predefined heuristics like class size or semantics as in [25,45], we use the online class-wise feature mean µ c and standard deviation σ c . We adopt the density-based [12] clustering algorithm using the following distance based on Fisher's ratio: The resulting super-groups are much more adaptive and meaningful than the predefined ones (e.g., rare, common, and frequent class groups), and facilitate better group-wise feature re-sampling. Please refer to the supplementary materials for the visualization of some super-groups of semantically similar classes.

Experiments
Datasets. Our experiments are conducted on two datasets: LVIS v1.0 [15] with 1203 categories and COCO-LT [42] with 80 categories. Both of them are designed for longtailed instance segmentation with highly class-imbalanced distribution. We choose LVIS v1.0 dataset over LVIS v0.5 since the former has more labeled data for meaningful evaluations and comparisons. LVIS v1.0 dataset defines three class groups: rare [1,10), common [10,100), and frequent [100, -) based on the number of images that contain at least one instance of the corresponding class. Similarly, COCO-LT dataset defines four class groups [1,20), [20,400), [400, 8000), [8000, -). We use the standard mean average precision (mAP) as the evaluation metric. Using this metric on different class groups can well characterize the long-tailed class performance. Following [15], we denote the corresponding performance metrics of rare, common and frequent class groups as AP r , AP c and AP f . Recently, Dave et al. [10] proposed that the mAP metric is sensitive to changes in cross-category ranking, and introduced two complementary metrics AP Fixed and AP Pool . We also report the performance of FASA under the AP Fixed and AP Pool metrics in the supplementary materials. Implementation details. Our implementation is based on the MMDetection [6] toolbox. We follow the same experimental setting as in [15,37] for fair comparison. Please refer to supplementary materials for other details.

Ablation Study on LVIS
We first perform ablation studies on the large-scale LVIS dataset. We report the validation performance to ablate the core modules of our FASA approach. Effectiveness of FA and FS. Table 1 verifies the critical roles of our adaptive modules for Feature Augmentation (FA) and Feature Sampling (FS). The baseline (top row) only performs repeat factor sampling [15] for real features, without using any FASA components. Our FA module (second row) significantly improves the performance, both for overall and rare classes. Our adaptive FS further boosts the performance, especially in the rare class groups (AP m r 12.7% → 17.8%) with other groups being competitive. The results suggest the effectiveness of both the FA and FS components in improving the training performance. Comparison with other augmentation methods. To further show the simplicity and efficacy of our method, we compare with the classic SMOTE method [4], MoEx [23], InstaBoost [13] and state-of-the-art methods [28,8,48] that are specially designed for the long-tailed setting. Since [28,8,48] only report results on face recognition and person Re-ID, while no public codes are available, we re-implemented them and optimized their parameters and performance for LVIS (see the supplementary materials for implementation details). Table 2 shows favorable results of our FASA when compared to others. Our gains are particularly apparent for AP r and AP c . Since SMOTE [4], MoEx [23] and Insta-Boost [13] are not directly designed for long-tailed setting, we verify that our FASA obtains a favorable performance over them. For more related feature augmentation approaches [8,28,48], we observe some limitations when transferring them into the instance segmentation task. Liu et al. [28] are based on margin-based face recognition loss such as ArcFace [11] that constrains the margin between each instance and its class anchor. Unfortunately, since the instance segmentation task has to deal with the special background class that no distinct anchor, the margin-based loss does not perform well on LVIS. In contrast, our approach is not restricted by the form of loss function. As for Chu et al. [8], the performance is limited by the small batch size of the instance segmentation task, which does not guarantee the selection of top confusing samples. Both [8] and [48] apply feature transfer from head to tail classes. They use a two-stage training pipeline that requires a pre-trained model to extract features. In contrast, our proposed FA method can be trained end-to-end, much faster than [8,48] and only incurs a small extra memory which is constant. Analysis of adaptive FS. Recall in Eq. (4), our class-wise feature sampling probabilities are adaptively adjusted by the scaling factor s c in an adaptive manner. Table 3 validates our design, showing that sampling is poor with a static sampling strategy. Specifically, we observe that s c = 5 works best for AP r , but is not optimal for AP f that needs s c = 1 instead. On the other hand, our adaptive FS tunes s c online to effectively re-balance the performance across categories. Fig. 3(a) depicts how the class-wise sampling probabilities change during training. Overall, the rare classes exhibit high sampling probabilities compared with the common and frequent classes. The rare-class sampling probabilities typically get increased to use more virtual features at the beginning. Then they gradually decrease to avoid over-fitting. Upon convergence, when the learning rate decreases, the sampling probabilities of rare categories see a bump to help with the corresponding classifier 'fine-tuning'. By contrast, the changes of sampling probabilities for the common and frequent classes are smaller. Fig. 3(b) further shows the dynamic changes of sampling probabilities within each class group.

Comparison with State of the Arts on LVIS
In this section, we evaluate the full FASA method against state-of-the-art methods on the LVIS v1.0 dataset. We consider the following representative methods: 1) Repeat Factor Sampling (RFS) [15] is an image-level data re-sampling technique. We use the same repeat factor 1e−3 as proposed in the original paper. 2) Equalization Loss (EQL) [37] is a loss re-weighting method that aims to ignore the harmful gradients from rare classes. 3) Classifier Re-training (cRT) [21] first uses random sampling for feature representation learning, and then re-trains the classifier with repeat factor sampling. 4) Balanced Group Softmax (BAGS) [25] first performs class grouping and then makes the classification loss relatively balanced across those class groups. Grouping simply relies on the class sizes that are independent of training dynamics and suboptimal. 5) Seesaw Loss [41] balances ratio of cumulative gradients for the positive samples and negative samples of different categories. Comparing using Mask R-CNN baseline. The aforementioned methods have shown solid improvements when combined with the Mask R-CNN baseline. Here we conduct a more comprehensive comparison with even stronger versions of these methods under different training schedules. Specifically, we experimented with the default 12 and 24 epochs schedules [6], as well as the decoupled two-stage training schedule [21,25]. In the first stage, we train the model for 12 epochs with standard random data sampling and cross-entropy loss. Then in the second stage, we finetune for 12 epochs using those advanced re-sampling or reweight approaches, such as RFS and BAGS. We refer to this schedule as '12+12'. We compare with all of these methods by plugging in our FASA module and comparing the performance difference. Table 4 summarizes the comparison results. Specifically, we repeat each experiment three times with different random seeds and report the mean value of results. When we combine our FASA with vanilla Mask R-CNN, we observe huge 3.3% gains in the overall metric AP. Our benefits are particularly evident in the rare class performance AP r , with gains of 9.0%. This validates FASA's superior ability to deal with long-tailed tasks. We also verify that the benefits of FASA remain consistent over multiple runs. When combined with stronger methods (RFS/EQL/cRT/BAGS/Seesaw) or training schedules, FASA can still obtain consistent improvements in overall AP, where the rare-class improvements in AP r dominate. Such gains come without compromising the common-and  Table 4: Comparing state-of-the-art methods with and without our FASA on the LVIS v1.0 validation dataset. We compare with the Mask R-CNN baseline, and state-of-the-art re-sampling approach Repeat Factor Sampling (RFS) [15], Equalization Loss (EQL) [37], Classifier Re-training (cRT) [21], Balanced Group Softmax (BAGS) [25] and Seesaw Loss [41]. The 'Uniform' method indicates random and uniform sample images. These methods are trained under different training schedules (24 or 12+12 epochs) using the public codes. All methods use ResNet-50 [17] as the backbone for fair comparison.  [47]) and advanced instance segmentation framework Cascade Mask R-CNN [2]. As shown in Table 5, the benefits of FASA still hold under different settings. On the validation set, FASA improves the performance of Mask R-CNN baseline (ResNet-101 backbone) by a large margin. We observe improvements in overall AP and rare-class AP r by 1.9% and 5.9% respectively. For the stronger ResNeXt101 backbone, we observe similar trend. FASA significantly improves the rare class performance AP r by 4.6%. For Cascade Mask R-CNN [2] framework and Seesaw loss [41], FASA offers consistent performance boost in rare classes while remaining strong on both common and frequent classes. Analyzing classifier weight norm. As discussed in [21,25], the weight norm of a classifier is correlated with the imbalanced learning performance, with the weight norms of tail classes often being much smaller on long-tailed data.  Entropy baseline and Seesaw Loss [41]. This partially explains why FASA greatly improves the rare-class performance of these methods.

Evaluation on COCO-LT
We evaluated FASA on COCO-LT [42] dataset to examine the generalizability of our approach. For a fair comparison, we follow the same experiment setting as SimCal [42]. There are two main differences compared to the implemen-   tation details we described in the LVIS dataset: (1) Singlescale training is used. We set the input image's short side to 800 pixels and do not use multi-scale jittering. (2) The bounding box head and mask head are class agnostic. Table 6 summarizes the results. The top row shows results of the Mask R-CNN [16] with ResNet50 [17] backbone and FPN [26] neck. The second row shows results obtained by the previous state-of-the-art method SimCal [42], which involves classification calibration training and then dual head inference. The third row shows results of the Mask R-CNN baseline augmented with our FASA. Note how FASA improves the performance of Mask R-CNN considerably, especially for those rare classes (AP m 1 and AP m 2 ). Overall, FASA performs better than SimCal [42] without any form of decoupled training.

Evaluation on CIFAR-LT-100
To demonstrate the generalizability of FASA in other domains, we further apply FASA on the long-tailed image classification task. We follow [3] to conduct experiments on the long-tailed CIFAR-100 dataset with a sub-stantial imbalance ratio of ρ = 100 (the ratio between sample sizes of the most frequent and least frequent class, We train for 200 epochs and use the top-1 accuracy as the evaluation metric. We compare FASA with two state-of-the-art feature-level augmentation approaches M2M [22] and Chu et al. [8]. When comparing with M2M [22], we use LDAM [3] loss as the baseline for a fair comparison. We use the same backbones of [22,8] to experiment under the same settings.
From Table 7, we observe that our FASA brings 1.7% accuracy improvement over the LDAM [3] baseline and is comparable with M2M [22]. When applied to the stronger baseline De-confound-TDE [38], FASA's benefits still hold. FASA is also better performing than Chu et al. [8] with the cross entropy loss. Other than that, our FASA is more timeefficient than M2M [22] and Chu et al. [8] because they need extra time for classifier pre-training while FASA can be applied online.

Conclusion
We have presented a simple yet effective method, Feature Augmentation and Sampling Adaptive (FASA), to address the class imbalance issue in the long-tailed instance segmentation task. FASA generates virtual features on the fly to provide more positive samples for rare categories, and leverages a loss-guided adaptive sampling scheme to avoid over-fitting. FASA achieves consistent gains on the Mask R-CNN baseline under different backbones, learning schedules, data samplers and loss functions, with minimal impact to the training efficiency. It shows large gains for rare classes without compromising the performance for common and frequent classes. Notably, as an orthogonal component, it improves other more recent methods such as RFS, EQL, cRT, BAGS and Seesaw. Compelling results are shown on two challenging instance segmentation datasets, LVIS v1.0 and COCO-LT, as well as an imbalanced image classification benchmark, CIFAR-LT-100.
In the supplementary materials, we discuss the ablation studies and implementation details that are not elaborated in the main paper. Section A highlights our FASA largely reduces the classification error. Section B analyzes some ablation studies of the adaptive feature sampling module. Section C validates the memory-and time-efficiency of FASA. Section D presents the clustering results used in the feature space. Section E provides the performance of FASA under the recently proposed AP Fixed and AP Pool metrics [10]. Section F reports our implementation details of feature augmentation methods. Section G compares FASA with another instance-level re-sampling based approach NMS resampling [45]. Section H shows the visualization result of FASA.

A. Error Analysis of Long-Tailed Instance Segmentation: Classification Error Dominates
In the main paper, we apply FASA only to the classification branch of Mask R-CNN. But why the choice? How does this simple mechanism impact performances of other branches like detection? To answer such questions, we need a detailed error analysis other than the default, single metric mean-average precision (mAP). We use the recent TIDE [1] toolbox to report six error metrics for long-tailed instance segmentation. Table 8 outlines the six error metrics on large-scale LVIS dataset. We see that for the Mask R-CNN baseline [16], classification error is the main bottleneck for long-tailed instance segmentation when compared to other error types, e.g., localization error. This explains our use of FASA to the classification branch of Mask R-CNN. Obviously, augmenting classification branch only will not incur a high cost. Performance-wise, we do see a significant reduction in classification error (from 25.10% to 20.74%) without deteriorating other errors much. With more budget, one could apply FASA to augment other branches of Mask R-CNN with the hope of more gains in all error metrics.

B.1. Initial Feature Sampling Probabilities
To initialize our adaptive virtual feature sampling process, we assigned class-wise sampling probabilities based on inverse class frequency. This scheme favors rare-class feature augmentation at the beginning, and does not rely on too many assumptions about skewed data distribution. Another sampling scheme that has minimal assumption of data distribution is based on uniform class distribution. Table 9 compares the two schemes empirically. We see that both the uniform initialization and the inverse class frequency initialization boost the performance compared with the no augmentation baseline. Overall, the inverse class frequency initialization approach achieves better overall mask mAP. The AP r and AP f results of uniform initialization are slightly worse than initialization based on inverse class frequency. So we use the initialization based on inverse class frequency by default for its effectiveness and simplicity.

B.2. Performance Metric for Sampling Adaptation
In the main paper, we propose a virtual feature sampling approach that is adapted to the validation loss rather than validation metric. This is to avoid the large computational cost from frequent metric evaluation on large-scale dataset. Concretely, evaluating the validation metric of mAP on the dataset takes nearly 45 minutes, which is very expensive if we were to conduct it in each epoch. To test how much we can gain from adapting to the true performance metric, we compare the use of the two supervisory signals on a smaller task. We choose the long-tailed image classification task on CIFAR-100-LT [3], a much smaller dataset than LVIS. Evaluating per-class accuracy is as efficient as evaluating the loss on CIFAR-100-LT validation set. Table 10 shows a marginal improvement from using the performance metric, which when translated to large-scale dataset, may not be worth the large cost for metric evaluation.

B.3. Group-wise vs. Class-wise Adaptation
By default, we adjust the feature sampling probability for each class group rather than each class. One of the reasons is that some classes may be missing for performance evaluation, e.g., on LVIS validation set. This makes it impossible for loss-adapted sampling probability adjustment for every class. But what if all classes are available for evaluation, appearing on both training and validation sets?
We again test on the CIFAR-100-LT [3] dataset that meets the requirement. In this case, class grouping is not a must anymore, and our goal is to see if group-wise sampling adaptation still holds its benefits over class-wise sampling adaptation. Table 11 gives a positive answer. We observe that group-wise sampling adaptation performs better and has a lower variance since it relies on the stabler groupwise loss average rather than the noisy per-class loss. Table 12 further validates the memory-and timeefficiency of our FASA approach. We see that FASA adds only a small amount of memory, which is used to maintain the online feature mean and variance of observed training samples. Thus the extra memory is constant and dependent only on the feature dimension. FASA is also found to incur a very small time cost.  [16] with and without FASA. We use the TIDE [1] toolbox and report the six error types (%) on LVIS v1.0 validation set: classification error (E cls ), location error (E loc ), both classification and location error (E both ), duplicate detection error (E dupe ), background error E bkg and missed ground truth error (Emiss). We observe the dominance of E cls for Mask R-CNN, and hence apply FASA only to the classification branch of Mask R-CNN at minimum cost. This leads to big improvements in E cls already. We expect more gains in E cls and other error metrics by augmenting other branches of Mask R-CNN if given more computational budget.  Figure 5: t-SNE [32] visualization of class groups. In the black dashed boxes, classes are often semantically related or visually similar. In the blue ellipse, we find classes that exhibit strong co-occurrence, e.g., between 'pillow' and 'bedspread'.

D. Visualizing Class Grouping Results
Recall our feature sampling probabilities are adjusted in a group-wise manner. The class groups are formed by the Mean-shift [7] clustering algorithm. Figure 5 presents some clustering results, where visually similar or semantically re-   lated classes stay close in the feature space (e.g., "hairnet" and "visor"). In addition, the co-occurrent classes also tend to stay close (e.g., "pillow" and "loveseat"). Intuitively, related classes are better suited to have their sampling probabilities adjusted together.

E. Evaluation with AP Fixed and AP Pool
The mean average precision metric (denotes as AP Old in the following) is the default evaluation metric for the instance segmentation task [27,15]. Recently, Dave et al. [10] argued that the AP Old metric is sensitive to changes in crosscategory ranking, and introduced two complementary metrics AP Fixed and AP Pool to replace AP Old for LVIS [15] dataset. Dave et al. [10] found that some methods improve AP Old but have less impact on AP Fixed and AP Pool . The AP Old metric limits the maximum detection results per image, resulting in cross-category competition. To address this issue, the AP Fixed metric limits the maximum detection results per class on the dataset instead. To highlight the score calibration property, Dave et al. [10] also proposed AP Pool metric that is class-agnostic and evaluates detection results across all categories together. Since AP Pool is classagnostic, the evaluation is influenced more heavily by frequent classes rather than rare classes.
We provide the experimental results of FASA under the AP Fixed and AP Pool in Table 13. We observe that FASA consistently boosts the performance under AP Fixed and AP Pool , especially for the rare categories. For AP Fixed , FASA improves overall AP and rare-class AP r by 1.1% / 1.7% respectively for Mask R-CNN and 1.3% / 2.4% for Cascade Mask R-CNN. For AP Pool , FASA still obtains 2.9% gains in AP r for Mask R-CNN and 2.0% for Cascade Mask R-CNN. Since AP Pool is mainly affected by the frequent class, the overall AP improvements are small. The AP Fixed and AP Pool results also demonstrate that FASA largely improves the performance of rare classes without compromising the common classes and frequent classes.

F. Implementation Details
Our implementation is based on the Mask R-CNN [16] backbone, which is the ImageNet pre-trained ResNet-50 [17] with a FPN [26] neck and a box head with two sibling fully connected layers for RoI classification and regression. We apply random horizontal image flipping and multiscale jittering with the smaller image sizes (640, 672, 704, 736, 768, 800) in all experiments. All models are trained with standard SGD on 8 NVIDIA V100 GPUs. We follow the default settings in MMDetection [6] to set other hyperparameters such as learning rates and training schedules.
Here we also describe the implementation details of the feature augmentation methods listed in Table 2 of the main paper. We detail the hyper-parameters of these approaches and our searched optimal choices in Table 14. SMOTE [4]. The SMOTE algorithm interpolates neighboring features (i.e., feature embeddings of region proposals) in the feature space. Specifically, for given features x i , we consider the k = 5 nearest neighbours {x j } based on cosine feature distance. Then we interpolate new features as: where λ is a random value in (0, 1]. MoEx [23]. Similar to SMOTE [4], the MoEx is also an interpolation-based augmentation method. As MoEx was developed for the image classification task, we transfer it to the instance segmentation task with the following modification: 1) we applied MoEx augmentation in the classifier of the Mask R-CNN [16] framework, 2) we searched the optimal value of parameters on the LVIS dataset and the results are shown in Table 14.
InstaBoost [13]. InstaBoost is an adaptive copy-and-paste FA method based on a location probability map. Since In-staBoost was already developed for the instance segmentation task, we use its default hyper-parameters.
Liu et al. [28]. Liu et al. [28] propose to transfer the angular distribution of face recognition loss such as CosFace [40] or ArcFace [11]. We select ArcFace for re-implementation: log e s(cos(θy+αy+ma)) e s(cos(θy+αy+ma)) + C j =y e s(cos(θj +αy)) (7) The symbol θ y refers to the angle between the input feature and the weight of the classifier. The symbol α y means the extra angular that transfers from head class to tail class. As shown in Eq (7), two parameters are involved: the symbol s means the scaling factor applied to logit, and the symbol m a refers to the angular margin. We tune these two parameters and show the results in Table 14.
We observed that since the instance segmentation task has to deal with the special background class, the marginbased ArcFace loss is unfortunately very sensitive to hyperparameter choices of m a . So margin-based augmentation [28] does not perform well on LVIS. Different from Liu et al. [28], our FASA is not limited to the form of loss functions. Chu et al. [8]. Chu et al. [8] mixed the class-specific features of each class and the corresponding class-generic features of 'confusing' classes to synthesize new data samples. The definitions of 'class-generic' and 'class-specific'  [2] with and without FASA under the recently proposed AP Fixed and AP Pool metric [10]. We report the results on the LVIS [15] validation set. AP, APr, APc and APf refer to the mask mAP metrics (%) for overall, rare, common and frequent class groups. The symbol AP Old refers to the standard mean average precision (mAP) metric. We observe FASA offers consistent performance boost under the AP Fixed and AP Pool , especially for the rare categories. All the models use the ResNet-101 [17] backbone and Repeat Factor Sampling (RFS) [15].   [48] αrecon Coefficient of the reconstruction loss 0.5 αreg Coefficient of the regularization loss 0.25 are based on the threshold masking of class activation map (CAM) [53]. For each real sample in the tail class, the authors sample N a images from its N f confusing classes.
During the re-implementation, we found that the difference in batch size between the classification and instance segmentation tasks limited the performance of Chu et al. [8] when transferred to the instance segmentation task. The classification task has a large batch size (e.g., 128) that can meet the demand of picking confusing categories (e.g., N a = N f = 3). However, instance segmentation models are limited by small batch size (e.g., 2) and there is no guarantee that the top confusing categories will appear in the same batch. Compared with Chu et al. [8], our FASA builds feature banks for each category to cache the features of the previous batch, thus getting rid of the small batch limitation.
Yin et al. [48]. Yin et al. [48] is a feature augmentation method designed for the face recognition task. A total of three loss functions are included: face classification loss L sf mx , reconstruction loss L recon and regularization loss L reg . The reconstruction loss L recon is critical to train the discriminative feature encoder and decoder. To transfer into the instance segmentation task, we apply the reconstruction loss to the feature embedding of each positive region proposal. Besides, Yin et al. [48] need a two-stage training pipeline. In the first stage, the authors fix the backbone and generate new feature samples to train the classifier. In the second stage, the authors fix the classifier and update the

G. Comparison with NMS Re-sampling [45]
NMS Re-sampling [45] is proposed to adjusting the NMS threshold for different categories during the training.
Specifically, the NMS thresholds for the frequent/common/rare categories are set as {0.7, 0.8, 0.9} with the increasing trend. Such a mechanism is beneficial to preserve more region proposals from the rare classes and suppress the number of proposals from frequent classes.
We compare the FASA with NMS Re-sampling in Table 15. The first line refers to the Mask R-CNN [16] baseline without any re-sampling or augmentation method.  From the second and the third line, we see that both FASA and NMS Re-sampling achieve better performance than the baseline method. FASA performs slightly better than NMS Re-sampling, especially for the rare classes. We believe such a performance gap is due to NMS Re-Sampling is mainly in adjusting the sampling weights of the current data samples, while FASA can further generate new virtual samplers. Also, the bottom results demonstrate that FASA as an orthogonal module can combine with NMS Re-sampling to further boost the performance.

H. Result Visualization
To better interpret the result, we show the segmentation results of the selected rare classes in Figure 6. We observe that without FASA, the prediction scores for rare classes are small or even missed. On the contrary, with the help of our FASA, the classification results of the rare classes become accurate.