Few-shot Open-set Recognition Using Background as Unknowns

Few-shot open-set recognition aims to classify both seen and novel images given only limited training data of seen classes. The challenge of this task is that the model is required not only to learn a discriminative classifier to classify the pre-defined classes with few training data but also to reject inputs from unseen classes that never appear at training time. In this paper, we propose to solve the problem from two novel aspects. First, instead of learning the decision boundaries between seen classes, as is done in standard close-set classification, we reserve space for unseen classes, such that images located in these areas are recognized as the unseen classes. Second, to effectively learn such decision boundaries, we propose to utilize the background features from seen classes. As these background regions do not significantly contribute to the decision of close-set classification, it is natural to use them as the pseudo unseen classes for classifier learning. Our extensive experiments show that our proposed method not only outperforms multiple baselines but also sets new state-of-the-art results on three popular benchmarks, namely tieredImageNet, miniImageNet, and Caltech-USCD Birds-200-2011 (CUB).

INTRODUCTION
Deep Neural Networks have made great success in computer vision tasks, such as image recognition, image segmentation, text understanding, and object detection [6,20,24,52,56]. Deep models based on Convolutional Neural Network (CNN) [11,13,17,35] architectures have been proved efficient in excavating representative information from data in many fields.
Supervised learning is one of the most dominant learning approaches for deep models, where the optimization of deep models can be achieved by feeding a large amount of training data to the models and utilizing their labels as the training supervision. However, supervised learning limits the model application to the pre-defined training classes, which is often described as the closeset learning task. It is problematic to directly deploy these models to the real world, as a variety of novel categories that are out of the training classes may exist.
As it is impossible to enumerate all the real-world classes for training, the learned model has to recognize inputs from new classes as one of the known classes, as illustrated in Fig. 1 (a), which hampers the applications of deep models in the real world.
Open-set recognition (OSR) is proposed, as a promising direction, to solve this kind of unknown class rejection problem. It aims to accurately classify the known classes and meanwhile detect the unknown classes. Many OSR methods make use of the large amount of training data to estimate the distributions of training classes, such that those out-of-distribution data can be recognized as unseen classes.
This has largely solved the open-set recognition problem in many tasks. However, there are still many application scenarios where these algorithms are not applicable. For example, in longtailed recognition, when the training data of some classes, e.g., rare animals, is not fairly enough, it is difficult to depict the distributions of these classes. Moreover, for tasks that rely on limited data to make predictions, such as few-shot learning, face verification, and person re-identification, these distribution-based methods would also fail.
To handle the problems in these special settings, a more challenging task, namely, Few-shot Open-Set Recognition (FSOSR) is proposed [21], which aims to classify inputs from both known classes and unknown classes with only a few labeled training data of seen classes available.
The goal of FSOSR model is twofold: to identify discriminative information from limited training data, and to have the ability to recognize unseen samples. The well-studied few-shot learning literature has well addressed the former problem, hence the main challenge of FSOSR is to detect unseen samples without harming the classification accuracy, which is the focus of this paper. The main difficulty is that the information from training data to use for obtaining discriminative classifiers of seen classes is already very limited. A question is then raised -where can we find the information of unseen classes? In this paper, we argue that the background regions provide valuable information that can be utilized to detect samples from novel classes. In closed-set recognition, a CNN is optimized to identify the discriminative information in the training samples, with the help of activation functions. As a result, only the discriminative parts in the images are kept as the representations of the data for classification, and the backgrounds are therefore ignored. Since these ignored backgrounds do not belong to any of these existing classes, we treat them as the pseudo unseen classes to learn a classifier that can classify both seen and unseen classes meanwhile. Instead of learning the decision boundaries of only seen classes, we reserve the space for unseen classes with the representations from the background, and we can easily conduct open-set classification by detecting samples located in these areas. Our motivation is illustrated in Fig. 1 (b). More specifically, we first initialize a class-mean classifier, where the prototypes, i.e., the weight vectors corresponding to different classes, are obtained by averaging embeddings of support samples from each class. We then randomly initialize a set of prototypes that correspond to unseen classes. Next, we use the background features as well as the features of support data to finetune the whole classifier. The labels of the background features are set as the nearest prototypes that correspond to unseen classes.
To accurately locate the background regions given limited training data, we further develop a Progressive Class Activation Mapping (ProCAM) module. Compared with the standard CAM [57], the Pro-CAM iteratively mines the class-related regions in images, which can better separate the foreground and background areas in the low-show case as shown in Fig. 2. We empirically show that our proposed ProCAM can not only generate better prototypes of unseen classes, but also regularize the learning of close-set classification.
Extensive experiments on multiple benchmark datasets have been conducted to verify the effectiveness of our proposed method. Our main contributions in this work can be summarized as follows.
• We solve the few-shot open-set recognition task as a multiclass classification problem, where a set of additional classes is added as unseen classes. • To obtain the data used for training the classifier, we propose to utilize the background regions of images from the seen classes. • We develop a Progressive CAM (ProCAM) module to iteratively seperate the class-related foreground regions and the background regions. • Experiments on the tieredImageNet, CUB, and miniImageNet datasets show that our method significantly outperforms the baselines and sets new state-of-the-art performance with remarkable advantages.

RELATED WORK 2.1 Few-shot learning
Few-shot learning focuses on training models with insufficient training samples to gain the ability of classifying unseen images [5,12,23,25,36,37,41,45,47,[49][50][51][53][54][55]. There are great diversities shown up in the literature of few-shot learning. The two main streams in the few-shot learning literature are optimization-based methods [4,7,18,32,34,37] and metric-based methods [12,36,43,46,47,50]. Optimization-based methods target at finding good initial parameters and adapting these parameters to new tasks effectively with low shot supports. Ravi and Larochelle [32] optimize the model parameters using LSTM in few-shot setting. Finn et al. [7] proposed Model-Agnostic Meta-learning (MAML) where a meta-learner is designed to adapt parameters for crossed tasks. TaskNorm [4] proposed a normalization technique for meta-learning tasks to fuse different normalization values for efficient optimization.
Our model is more related to metric-based methods, which aim to distinguish different classes of instances with distance metrics in their feature space. MatchNet [43] introduces cosine similarity to measure data distance and compute the similarity scores. ProtoNet [36] proposes the negative L2 distance to measure the difference between the class prototypes and the query vector. Deep-EMD [50] applies Earth Mover's Distance (EMD) as their distance metric. Recent methods transform the class prototypes to taskspecific prototypes with task-adaptive weights [19] or transformer [47]. The limitation of the existing FSL methods is that it only guarantees the good performance under the closed-set settings. The query set and the support set are in the same classes. FSOSR extends the FSL task with the unseen detection capability. We inject the additional background classes into conventional FSL methods and get a better detection capability.

Open-set recognition
Open-set recognition addresses the classification model with the detection capability of unseen samples [2,15,16,28,[38][39][40]48]. It aims to reject unseen samples using the open-set classifier and classify the seen samples. The model cannot see the training samples for open-set data, and can only use the training data from the close-set. Bendale et al. [2] train a close-set classification network first and then endow the trained network with open-set discrimination using Extreme Value Theory. Gopenmax [8] synthesizes fake open data during training. Neal et al. [28] propose to generate counterfactual images for the unknown class and train an (N+1)-class classifier to recognize the N known classes and 1 unknown class. Zhou et al. [58] also introduced the (N+1)-way classifier to solve the openset recognition problem, but they synthesize fake open data using manifold mixup [42] techniques, while we utilize image background. Recently, GAN-based methods gain popularity [1,8,16,28,29,39]. C2AE [29] and CGDL [39] use all the training samples to train an autoencoder and detect unseen samples by the large reconstruction error. Perera et al. [31] propose reconstructing the image along the channel dimension. The reconstruction of the unseen class samples is easy to fail which will lead to a lower classification probability for the unseen class. The normal OSR methods all require rich training data to train the unseen sample detector [2,28,29,31,39], which is hard to train in the FSL situation and getting poor performance.

Few-shot open-set recognition
FSOSR is recently proposed to address the open set recognition tasks in the few-shot inputs scenario [14,21]. PEELER [21] proposes the FSOSR task and solves the problem using the meta-learning framework and Mahalanobis distance to classify the known samples.
They introduce an open-set loss using a negative entropy loss to maximize the entropy for seen classes. SnaTCHer [14] uses the pre-train model with the transformer from FEAT [47] and needs to reuse the transformer to remap all the known and unknown query samples and a set of prototypes to a new space to calculate the distance. Our method is a metric-based FSOSR approach which is the same as PEELER and SnaTCHer, but we do not need the transformer module. PEELER [21] uses Mahalanobis distance. It needs to calculate the time-consuming covariance matrix. We follow MatchNet [43] to use cosine distance which is simple and effective.
FSOSR is different from the Generalized zero-shot learning [9,22] (GZSL). Although both do not have support samples for new classes, GZSL provides the semantic information for the target classes. FSOSR targets rejecting unseen new classes, the class information is unknown. As for GZSL, it is known information with class-level semantic description and target to classify target class samples to the correct classes.

PROBLEM SET-UP
FSOSR aims to design a machine learning algorithm that can reject unseen classes from only a few seen training examples and classify the seen classes samples correctly. 1 The typical setup for FSOSR follows episodic training and testing schema where batched tasks are sampled for training or testing. Specifically, an N-way K-shot FSOSR problem contains the support set and the query set for each episode.
For support set D = { , } =1 , where ∈ X and ∈ Y . The data from the support set X are from the classes seen with the label Y . The query set for FSOSR is different from FSL, D = D ∪ D , where D is the known query set and D is the unknown query set. The known query set are samples from the seen classes with the same label space  We introduce extra background classes to reserve space for unknown classes. Our background classifier uses background features as unknowns to optimize the classifiers. We use a ProCAM module to collect multiple normalized class activation maps to find the foreground mask. Query embedding can be a known query or an unknown query, which is encoded with the same backbone. We use cosine similarity to obtain the class score.
class. The unknown query is sample from the unseen class with There is no overlap for the seen labels and unseen Y ∩ Y = .
An algorithm fits model ( ) is defined as ( ) = W Φ( ), where Φ(·) : R → R is the embedding function and W ∈ R × is the Fully Connected(FC) layer to classify seen classes. We denote the classifier for class as w : W = [w 1 , · · · , w ].

METHOD
In this section, we introduce our framework for few-shot Openset Recognition. We first describe our extra background classifier method for clustering the unknown samples and separate the known classes in Section 4.1. Then we present our proposed background feature as unknowns schema to train the extra background classifiers in Section 4.2. The overview of the entire training pipeline is shown in Fig. 3.

Extra background classifier
The purpose of Few-shot Open Set Recognition (OSR) is to separate unknown samples from known samples. The known samples need to do close-set classification in further steps. The difficulty lies in that unknown samples are incorrectly classified as known samples if the classifier only contains known classes. For a CNN classifier with classes, it will divide the embedding space into parts. The class boundaries are determined by the discriminative information in the classifiers. Unknown samples projecting to the same embedding space will fall into known classes space and classify wrongly with such kinds of classifiers. Therefore, we propose to reserve space for unknown classes, which is simple and straightforward by adding extra class classifiers. We get the hint from a special class in segmentation problems, the background class, which may contain a lot of unknown information compared to the foreground known classes. We name our extra class as background class for easy understanding. Let W ∈ R × be the parameter matrix for the additional background classifier. It has the same number of feature channels as the known classes classifier W ∈ R × . is the number of known classes and is the number of background classes we pre-set to cluster possible unknown samples. The final FC of our network becomes W = [W , W ] ∈ R ×( + ) .
We employ the episodic training approach [36] to train our model. Each mini-batch of training will mimic an FSOSR task, the classifier weight for each known class can be seen as the average embedding for each class .
We introduced the background features as unknowns method to train our background class classifier using the background features of the support data. The details are provided in Section 4. 2 We use the cosine distance as our distance metric to measure the similarity between the query embedding Φ(X ) and the final FC of our network W.
The highest similarity of Φ( ) to the classifier weight w define the prediction of belong to . The loss can be expressed as:  whereˆ= argmax{sim⟨W , Φ( )⟩} + | | is the background class label with the maximum similarity the unknown query belongs to. The second term matches the unknown query to the reserved background classes space.

Background feature as unknowns
FSOSR lacks open-set data to train the model. The existing OSR method normally uses GAN to generate pseudo-samples for openset data, which is time-consuming and requires an extra branch of the generator network. We notice that the background of support data usually contains extra information and has a large difference from the foreground known class data. We can use background features as unknowns to train our extra background class and optimize the known classifier. Figure 4 shows the details of the classifier training. The training input is the support feature and the background feature of the support. The target label for the support feature is the known class label. We assign a background class label to the background feature. The backward propagation will only fine-tune the classifiers through the following loss.
The background feature can be extracted from the class activation map of different known classes. We proposed a ProCAM module that utilized multiple normalized class activation maps to mine class-related regions and get better background features.

ProCAM
The ProCAM module is developed to apply CAM on the original support features multiple times to figure out the previously uncovered known class regions. The maps found multiple times can be superimposed to get more complete background area. This section introduces details on how to generate multiple class activation maps. A class activation map for a specific category will highlight the discriminative regions for that category. For a given input image with labeled class , let ℎ 0 ( , ) represent the activation of the image in spatial location ( , ) in the last convolutional layer before average pooling. The CAM for class can be calculated through the following equation: where w is the classifier weight for class . We scale 0 ( ) to [0, 1] using the Min-Max normalization.
The first background feature map is getting through ℎ 1 = ℎ 0 × (1 − ( 0 ( ))). We regard the first background as the input of the CAM equation again, the CAM function will highlight more discriminative regions to the class .
By repeating this operation for times can give us different CAMs [ 0 , 1 , · · · , ( )]. We sum all these CAMs up and normalize them again to get the final ProCAM.
The final background feature map ℎ = ℎ 0 (1 − ), where ℎ 0 is the original feature map. We improve ProCAM using softmax normalization to emphasize the high response regions and squeeze the low response regions to get a better mask for the foreground area.

EXPERIMENT 5.1 Datasets
miniImageNet [43]: It is a popular benchmark for few-shot learning with 100 classes and 600 images in each class. [21] proposed to use this dataset for the FSOSR task. Following the conventional splitting [43] of meta-training(64), meta-validation (16), and metatesting (20) for FSL, the few-shot open-set problem samples training episodes from the 64 meta-training classes, and testing episodes from the 20 meta-testing classes.
tieredImageNet [33]: It is another popular FSL benchmark with more classes, which includes 608 classes from 34 super-classes. We follow [14] to split training, validation, and testing into 351, 97, and 160, respectively.
Caltech-UCSD Birds-200-2011 [44]: CUB is a fine-grained bird classification dataset with 200 categories and in total 11,788 images. Under the 5-way FSOSR setting, we randomly sampled 5 classes as the closes set and 5 classes as the open sets from 100 meta-training classes to form the training episodes. The validation episodes and testing episodes were sampled from the rest 50 meta-validation classes and 50 meta-testing classes.

Implementation Details
Following [14], we use the same network backbone Resnet12 for all datasets, where the last layer before the classifier contains 640 dimensions. We built our project using PyTorch [30]. The model was pre-trained with batch size of 128 for 500 epochs and optimized  ( ) ← Get CAM from ℎ ( ) and with eq. 5; 8: ( ( )) ← Get normalized CAM using eq. 6; loss3 ← Compute loss with L 3 ( ,ˆ), 16: loss4 ← Compute loss with L 4 ( ,ˆ), 17: loss=loss3+ loss4 ← get total loss, 18: {W ′ , W ′ } ← Update classifier with SGD 19: end while using SGD with momentum. The pre-training learning rate is adjusted using cosine annealing from 0.1 to 0 in a cycle of 100 epochs. Standard data augmentation with random crops, random color jitters, and random horizental flips are applied during pre-training. Manifold-mixup in conjunction with self-supervision [27] is also used to make backbone more robust. Our baseline is based on the first FSOSR paper [21] using meta-learning strategy to finetune the backbone episodically. We use cosine similarity as our distance metric and fine-tune the network in 20,000 iterations with an initial learning rate 0.0002. Ours ProCAM and softmax ProCAM are finetuning the extra background classifier with 0.002 initial learning rate for 20 epochs. The loss weight is set to 0.05 for miniImageNet, CUB, and 0.005 for tieredImageNet. We used the validation data to  select the best model and evaluated the test data for 600 episodes to minimize uncertainty. Evaluation Protocol. We follow [14,21] to evaluate the model in two metrics, the known class classification accuracy (acc) and the area under the ROC curve (AUROC) for the unknown class detection. The known class accuracy takes the known query ground truth label and checks the model corrected classification ratio.  Table 3: Comparison with the State-of-the-art methods. Average closed-set classification accuracies (%) and average unknown detection AUROCs (%) over 600 episodes for MiniImageNet and TieredImageNet. PEELER* is quoted from the paper, which has a ResNet-18 backbone.  Table 4: Comparison with the State-of-the-art methods. Average closed-set classification accuracies (%) and average unknown detection AUROCs (%) over 600 episodes for CUB.
is plotted based on the different thresholds for the true positive rate versus the false positive rate. The higher AUROC indicates the better unseen detection capability.

Analysis
We evaluate the effectiveness of different initialization methods for the background class in Table. 2. For each initialization method, we compare the different number of fine-tune epochs for the background class classifier learning. The random initialization method uses the Kaiming uniform initialization [10]. The 'avg' method initializes the background classifier weight with the average background feature maps. The 'global' method uses a global FC for background class across all the training and testing episodes. The results show that random initialization for every episode and fine-tuning with 20 epochs have a better result compared to the other methods. Fig. 5 shows the accuracy and AUROC change w.r.t. the number of repeating ProCAMs and the number of additional background classes. There is an increasing trend for accuracy and AUROC when the number of repetitions increases and reaches a maximum at = 4. Increasing the number of extra background classes does not affect the accuracy too much, but the AUROC changes a lot. The highest AUROC for miniImageNet 5-shot is adding 2 extra background classes. known class prototypes class proto after fine-tune known class support known query unknown query background features Figure 6: T-SNE [26] visualization of data embeddings and classifiers before (left) and after (right) fine-tune with background features as unknowns. Crosses with different colors represent query data points from different known classes. The pink plus represents unknown query. The blue squares and triangles indicate the classifier prototypes before and after fine-tuning, respectively. Our module reserve the space for the background classifier as shown in the green area.
We also plot the t-SNE figure [26] to see the update of the feature space for the background class. Fig. 6 shows that our proposed background classifier helps reserve space for unknown samples and provides a direction for the unknown class to cluster to the extra background class centers.
We provide a visualization of our ProCAM and softmax ProCAM, and compared them with the original CAM in Fig. 7. Our methods generate more accurate foreground masks, which cover all the important features in the known classes. The background feature based on our mask contains less discriminative information for seen class and can provide good background data to train our extra background classes. The background feature data is also helpful for pushing the boundary of known classes. Softmax ProCAM is similar to the ProCAM results, the background and foreground boundaries are more clear than ProCAM.
Ablation study. We perform our ablation study on the 5-way 5-shot setting for tieredImageNet to analyze the effectiveness of different components in 1. The base method is the testing result   [57]. Ours can generate more accurate foreground masks and get better background features that contain less information from the known classes. Our Softmax ProCAM generates better background features through the well-defined boundaries (The background regions heatmap is darker compared to the others).
for the pre-train model under FSOSR setting directly. The method 'meta_cos' employs the meta-learning and cosine distance and gets a better result than the pre-training model. We apply the vanilla class activation map to the previous 'meta_cos' method and we can get higher accuracy with better AUROC for this 'meta_cam' model. The previous CAM method only uses the background feature to fine-tune the background class classifiers, we try to fine-tune all the FCs using both the support feature and the background feature in 'meta_cam_ft'. The result provides around 0.6% of the increase in the accuracy, which indicates that the background features can help push the boundaries of the known classes and train better background class classifiers. We optimize the fine-tune of all FCs with better background feature maps. The results show an increasing trend in known class accuracy and unknown class AUROC.

Comparison with the State-of-the-Art Methods
Finally, we conduct comparison of our performance with the stateof-the-art results on three benchmarks: miniImageNet, tieredIm-ageNet, and CUB. We choose the state-of-the-art FSL methods ProtoNet [36], FEAT [47], and ASY [3] as the first group of models. In the second group, we select two OSR methods (NN [15], Open-Max [2]). The third group is our baseline with mete-learning and cosine classifier and the current existing FSOSR methods PEELER and SnaTCHer [14,21]. Because the existing FSL and OSR methods are not designed for FSOSR tasks, we follow SnaTCHer [14] to slightly modify them for comparison. We show the detailed numbers for miniImageNet and tiered-ImageNet in Table 3. The comparison for CUB is in Table 4. Our model has the highest AUROC of all the methods. The accuracy of the known classes is also competitive among all the datasets. In particular, our AUROC outperforms the state-of-the-art result by 3.14% on miniImageNet 5-way 1-shot. Our accuracy increased 1.64% for the same dataset 5-shot. For tieredImagenet, the 1-shot accuracy is slightly lower maybe due to the lack of support background data, but the AUROC still improved around 1.27%. The accuracy and AUROC on CUB outperform the state-of-the-art method by 10.56% and 11.5%, respectively.

CONCLUSION
In this paper, we solve the few-shot open-set recognition problems from two aspects. We first introduced extra background class classifiers to reserve space for unknown samples, which use cosine distance to determine the space for each class. Then, we propose a learning paradigm to utilize the background features as unknowns to train the extra background class classifiers and fine-tune the known classes' FCs. To generate a better mask for the foreground objects and get a better background feature, we propose a ProCAM module to utilize multiple class activation maps. We optimize our ProCAM module using softmax to separate the foreground part and generate clear boundaries of the background. Experiments on three datasets show that our method significantly outperforms the baselines and the state-of-the-art approaches.


Design of Convolutional Extreme Learning Machines for Vision-Based Navigation Around Small Bodies

Deep learning architectures such as convolutional neural networks are the standard in computer vision for image processing tasks. Their accuracy however often comes at the cost of long and computationally expensive training, the need for large annotated datasets, and extensive hyper-parameter searches. On the other hand, a different method known as convolutional extreme learning machine has shown the potential to perform equally with a dramatic decrease in training time. Space imagery, especially about small bodies, could be well suited for this method. In this work, convolutional extreme learning machine architectures are designed and tested against their deep-learning counterparts. Because of the relatively fast training time of the former, convolutional extreme learning machine architectures enable efficient exploration of the architecture design space, which would have been impractical with the latter, introducing a methodology for an efficient design of a neural network architecture for computer vision tasks. Also, the coupling between the image processing method and labeling strategy is investigated and demonstrated to play a major role when considering vision-based navigation around small bodies.

surface variations are only due to morphological characteristics (i.e. craters, boulders, etc.) which only vary under illumination conditions. This domain is fundamentally simpler than that of typical computer vision applications in urban environments, which need to account for a large variety of commonly used objects, where deep architectures are the state of the art [2]. Moreover, previous findings in [9] hint to the fact that the filtering capabilities CNNs on images of a small body are critical in pushing the performance compared to traditional methods.
Pivoting on these prior works and the existing gaps in the literature, in this paper the authors attempt to answer three fundamental questions, namely: 1) Although CNN are superior over complex scenery, can simpler methods perform better when it comes to analyzing images of small bodies?; 2) As suggested in [6], can a methodology be developed to bootstrap the training of CNN exploiting the capability of CELM to identify the most promising architectures?; 3) Which is the best labeling strategy and reference frame to be used when considering a visual-based navigation application around a small body?
In this work, CELM is thus investigated as a possible alternative to CNN for autonomous vision-based navigation systems around small bodies. This is done with an extensive analysis considering 4 different small body shapes and 5 different labeling strategies, for a total of 20 scenarios, each of which is examined with 4 different IP methods, resulting in tens of thousands of different architectures explored. Such efficient exploration of the architecture design space is possible thanks to the extremely fast training time of CELM, which is orders of magnitude faster than that of CNN [7]. It is also demonstrated that the labeling strategy and reference frame play a crucial role during training and significantly affect the performance of the methods considered. We believe that CELM can represent, with the proper labeling strategy, a promising alternative in space imagery, especially the one related to small bodies.
The rest of the paper is organized as follows. In Section II the three pillars sustaining the methodology are discussed in detail. These are the dataset generation, the preprocessing pipeline of the image-label pairs, and the description of the IP methods developed and used in this paper. The performance of these methods is then compared extensively in Section III while conclusions, and future works are discussed in Section IV.

A. Dataset generation
In this section, the dataset generation procedure is illustrated. Four small bodies are considered: 65803 Didymos (D), 103P/Hartley (H), 21 Lutetia (L) and 67P/Churyumov-Gerasimenko (P). Using the same methodology illustrated in [10], artificial morphological features such as boulders and craters are inserted into the shape models of these bodies.
For each body, a total of 17500 grayscale images are rendered in Blender * . These are split into training, validation, and test sets respectively made of 7500, 5000, and 5000 images. As illustrated in Fig. 1, these sets are randomly distributed across a cloud of points around the body in what is defined as the reference frame. This reference frame is an inertially fixed reference frame, centered on the Center of Mass (CoM) of the body, with the -axis oriented towards the projection of the Sun in the body equatorial plane and the -axis as the north pole of the body. Another reference frame that is used in this paper is the one, whose axes are fixed to the body surface and are obtained from a rigid rotation around the -axis of the reference frame. The points in Fig. 1  Since each dataset is composed of the same set of points across different bodies, uniform scaling is applied to each one to make sure that different body sizes are all filling the Field Of View (FOV) around 5 km. To generate the 8-bit grayscale images, a camera with a 10 ×10 deg FOV and a sensor of 1024 ×1024 pixel is considered.
All images are rendered assuming ideal pointing towards the CoM. From an optical-navigation point of view, images can be represented by different sets of labels. Five of these are considered in this work to investigate the influence of this choice on the performance of the IP methods. The first one is represented by optical observables linked with geometrical quantities directly detectable from images. In this work, these are: where is the difference in pixels between the Center of Brightness (CoB) and Center of Figure (CoF) of the body projected in the image plane and is the range from the CoM. These quantities can be used to generate a position estimate in the camera frame, which can be transformed in or by simulating on-board attitude determination from a star-tracker alongside the assumption of knowledge of the rigid rotation between the inertially fixed reference frames used by star-tracker and the or frames. The second and third sets are represented by the spacecraft position respectively in spherical and cartesian coordinates in the frame. Similarly, the fourth and fifth sets are represented by the same coordinates in the frame.
By combining the 4 different small bodies with these 5 different labeling strategies, a total of 20 cases are considered in this paper. The notation used to distinguish between these cases is summarized in Tab. 1. Table 1 Notation used for the datasets used in this work.

B. Preprocessing
After the dataset generation step, a total of 70000, 1024 × 1024 grayscale images with ideal pointing towards the CoM of each body are obtained. These images, however, cannot be used directly as input of the IP methods considered in this work for three main reasons.
First, the original image resolution is too high. Due to hardware limitations, image size needs to be reduced. This is typical of the data-driven IP methods used, which could encounter memory or processing saturation issues if working with images at native resolutions. Second, the ideal pointing assumed during rendering simplifies image generation but causes poor variability of the input-label relationship, which can cause poor generalization capability of the IP methods. Third, rendered images are ideal, thus far from realistic camera acquisitions.
All three issues are addressed together in a unique preprocessing pipeline which is a novel contribution of this work.
The pipeline transforms an image and its associated labels from a geometrical and rendering space referred to as S 0 into a new space S 2 that can be efficiently used by data-driven IP algorithms. A sketch of the pipeline is illustrated in Fig. 2 for clarity. The starting point of the pipeline is the image rendered in Blender, which by definition belongs to S 0 space. The image is binarized using the Otsu method [11] and a simple blob analysis is performed to compute the CoB (represented by the yellow point in Fig. 2) and the bounding box Γ (defined by two corner coordinates Γ 1 , Γ 2 , its width Γ 3 , and height Γ 4 ). All these quantities are computed in the frame [12], which is considered in the image plane as centered on the top-left corner of the image.
Cropping is performed around Γ to transform the image to a Γ 3 × Γ 4 snippet. Because the IP methods considered in this work assume fixed-size inputs, the variable-size snippet needs to be transformed into a fixed-size one.
The upper closest possible value between the largest size of Γ 3 × Γ 4 and 128, 256, 512, or 1024 is computed. Once the target size is determined, padding is performed using two parameters, , and , which are randomly generated.
These represent the necessary integer lengths that need to be added to Γ and are chosen respectively as 0 ≤ ≤ − Γ 3 and 0 ≤ ≤ − Γ 4 . After random padding, the image is said to be transformed in S 1 space. Note that the padding is performed by re-using the same pixels from the image defined in S 0 or by introducing zero-pixels values (using the nearest criteria) whenever this is not possible. When passing from S 0 to S 1 also the labels linked to the image need to be transformed, resulting in a change in the and coordinates: while the other labels remain unchanged: Noise is then added to the image, which is resized to a 128 × 128 resolution, transforming it in S 2 space. The noise step is optional for images in the datasets, while in an operational scenario it would be avoided. When passing from S 1 to S 2 the labels are transformed as follows: while 1 and 2 remain unchanged:

. Combining Equation 2 with Equation 3, Equation 4, and
Equation 5 it is possible to transform the image-label pairs for all the datasets in Tab. 1. All the IP methods used in this work use the image-label pairs in S 2 space, which need transformation to S 0 for on-board usage. The methods working on the datasets labeled with cartesian and polar coordinates either in the or reference frames generate directly a position estimate as: where reflects the reference frame used, and Ω is the transformation function from spherical to cartesian coordinates.
On the other hand, the methods working with the ( , ) labels need intermediate steps to generate a position estimate.
In inference, these methods generate the following optical observables: where S 2 and S 2 are output of the IP methods while S 0 and are parameters computed during the image preprocessing algorithm. In this work, the latter quantities are computed offline, however, it is noted that the algorithm can work also online during inference to prepare any incoming image with the proper format for the application of the IP method. The observable vector ,S 0 is transformed from the reference frame which express pixel coordinates on the image, to the Image Plane (ImP) reference frame using the inverse of the camera calibration matrix −1 [12]: The vector is then transformed into a Line of Sight (LoS) vector in the reference frame. Using the attitude quaternion of the spacecraft (which is assumed to be known from attitude determination from a Star-tracker) and assuming to know the rigid rotation between the inertial reference frame used by the star-tracker and a known asteroid frame, this LoS is transformed with the use of S 0 into a position estimate in reference frame: In this work, the frame is used for simplicity and no error is simulated on the attitude quaternion.

C. Image Processing
In this section, the IP methods are described in detail with a standardized notation, illustrated in the architecture outlined in Fig. 3. The input is represented by a 128 × 128 × tensor while the output is a vector or matrix whose elements represent the specific labels associated with each image or tensor. To simplify the discussion, the architecture is conceptually divided into two main portions: the encoder and the head.
In the first one, a hierarchical sequence of convolutions, activation functions, and pooling operations are performed to extract spatial information which is synthesized in the last layer in a latent tensor represented in orange in Fig. 3.
Such 3D tensor is flattened into a 2D vector referred to as the fully connected layer which constitutes the first layer of the head of the architecture. In this portion, several hidden layers may or may not be introduced before the output layer. A network is therefore established to map the connection between the neurons from the fully connected layer, which embeds spatial information extracted from the image in the latent space, and the output layer, which expresses the desired output label of the architecture.
In this work, convolutional layers are driven by weights and biases that only influence the kernels used in the convolution operations. On the other hand, the weights and biases of the head are representing the influence of the neuron connections between the fully connected and output layer. In this work, it is important to consider these two sets of weight and biases separately, since depending on the training strategy, they are handled differently. The former will be referred to as and , the latter to and 0 . The set of weights and biases of the entire architecture defines the set of parameters referred to as = ( , , , 0 ). Another parameter used to describe the architecture is represented by which reflects the set of architectural and training choices often referred to as hyper-parameters.
Within this context, to find an IP method which performs well means to find the best desirable sets of and which optimize predefined metrics. Training an architecture can be reduced to a parameter estimation problem with a double nested optimization loop. During training, both architectural and training choices (global parameters) and inner weights and biases (local parameters) need to be found out through data. When these are found, they are frozen and at inference the architecture can be considered as a function (parameterized by and ) on the input tensor that generate an output vector : Now that a shared nomenclature is established, the characteristics of the four IP methods are described in details.

Convolutional Neural Network
In a CNN the architecture in Fig. 3 is treated as a neural network whose local parameters are found using GD All the GD-based methods are structured in a two-phase process. First, a forward pass is executed by a specific network defined by to generate an output. The comparison between such output and its ground-truth value is quantified by a loss metric, which in turn is fundamental in evaluating the performance of the network as well as to determine the necessary adjustments to for improvement. These are determined from the computation of the gradient, that is estimated in a backward pass in the network followed by an update of to be used in the next iteration. These two passages, especially the last, are computationally expensive, making the training a laborious procedure. A possible way to reduce the training time is to avoid the backward pass, that also require a different optimization scheme.

Convolutional Extreme Learning Machine
A CELM removes the need for a backward pass by solving the optimization problem on the connections between the last hidden layer and the output layer of the architecture.
In this work, the architecture and initialization of CELM are the same as the one of a CNN. However, greater importance is given in the CELM on the random distribution of the eightscnn and of the kernels and of 0 . Once they are randomly initialized when the architecture is generated, they are frozen and are not changed during training. This is not the same for eightscelm, that are the only ones changed during training. Given a set of true input-output samples ( , ), the forward pass of the input into the network generates a hidden layer output matrix right before the output layer: . . .

( )
where is the dimension of the hidden layer before the output one, represented by the number of neurons in the layer.
The training data target matrix is then defined as: To find the best set of weights that matches the matrix , the following optimization problem shall be solved [7]: Minimize : that is a regularized least square that depends on the coefficient. The inclusion of first term regarding the minimization of the weights vector increases the stability and improves generalization capabilities [4]. The minimization problem can be solved in an efficient way as: Since the remaining weights and biases ( , , and 0 ), are randomly fixed at initialization and are never changed, there is no need for a backward pass. Only the forward pass and the LS problem need to be processed, which makes the training of a CELM architecture much faster than the one of a CNN. Note that in this work 0 is not considered in the CELM architecture.

Hybrid Convolutional Extreme Learning Machine
By combining the design of the CNN and the one of the CELM, the capability of a hybrid IP method is also investigated.
In the Hybrid Convolutional Extreme Learning Machine (HCELM), transfer learning is used to get the weights and biases of the kernels from the encoder of a previously trained CNN architecture, while the head is trained using the LS method, as in the CELM paradigm. Two IP methods have been designed with this strategy: one that uses the encoder from the best CNN architecture for that dataset (referred to HCELM), and one that uses the encoder from the best CNN architecture for that reference frame (referred to Hybrid Convolutional Extreme Learning Machine 3 (HCELM 3 )). In

Training
In this section, the strategy to train the IP methods is described. The architecture of each method is generated following a rigid procedural methodology to define the parameters of Θ. A schematic of the procedure is illustrated in Fig. 5.
To begin with, a great number of CELM architectures are initialized following the same rules for each dataset. The encoder of each architecture is designed with a hierarchical structure [6]. While going deeper from input to the fully connected layer, the starting 128 × 128 × 1 tensor is squeezed; its size is halved as function of the depth level as = 2 7− while its depth is doubled starting from an arbitrary value of 16. Each depth level is made up of the consecutive application of a convolution, activation function, and pooling operation.
The convolutions are performed with 3 × 3 kernels while the number of kernels used at each depth is set to increase exponentially as: The number of weights and biases at each depth level of the encoder can thus be determined: Store β(C) on the depth level as = 2 , the number of and 0 of the head is computed as:

CNN -Training and Validation
where 0 is the number of neurons composing the output. For improved handling of the labels, whenever the ( 1 , 2 , ) labels are considered the azimuth angle 1 is transformed in the adimensional ( 1 , 1 ) pair. 0 is thus equal to 3 in all datasets but those for which spherical coordinates are used as labels, for which 0 = 4. An example of a 5 layers architecture is illustrated in Tab. 2 with a parameter count divided per depth. Having defined a procedural set of rules to generate each architecture, these are generated with the hyper-parameters Θ summarized in Tab. 3.
By considering the possible combination between , , , and a total of 120 different CELM architectures are considered. For each kernel distribution, a random initialization is executed 3 times, meaning that a total of 360 CELM networks are generated for each dataset in Tab. 1.
Once the forward pass of the CELM is executed, for each architecture the LS optimization problem is run 9 different times during training of the CELM with different regularization terms . The training set is used to determine all possible values of depending on , while the validation set is used to determine the best value of . The combination of determined from the training set and determined from the validation set is used in inference on the test set to produce the estimated labels. Because the training time of CELM is orders of magnitude faster than the one of the CNN (∼ 1 compared to ∼ 600 , on average), with the use of CELM networks it is possible to explore the architecture space to find those that seems inherently more suitable for the task at hand. As suggested in [6], a great portion of the performance of the CNN seems to be given by the architectural choices, which are often neglected nor sufficiently  For each dataset, the best CELM network is defined as the one achieving the minimum positioning error (defined in Sec. III) on the test set while its parameters are saved.
For each dataset, a CNN architecture is then initialized with the hyper-parameters found from the CELM architectures.
This CNN is then trained by varying and and performing 3 runs for each combination, for a total of 45 cases. For each of them, the CNN is trained for 300 epochs while the best value on the validation loss is used to instantiate the weights and biases of the best possible realization of the CNN.
In this way, we provide the CNN with an architecture that has proven to work properly even with random kernels and investigate whether the re-arrangement of the weights and biases of these kernels together with the ones in the head portion can further improve the baseline performances by extracting additional information or not. The set of for the best CELM and CNN architectures is summarized in Tab frame. This means that in the first case 20 encoders are used, one for each architecture, then 3 encoders are used, each of them being the representative of the CNN architecture working best in that reference frame.
All IP methods are considered to work with normalized input and output, which has been observed to improve the overall performance. The labels are normalized over the maximum and minimum values found in each training set.
While CELM are trained with the entire dataset at once, CNN architectures are trained with batches of images. In the

III. Results
The four IP methods trained over all datasets of Tab where ,S 0 , ,S 0 are respectively the estimated and true position in the reference frame while S 0 is the true range from the body CoM, all evaluated in S 0 . Also, for the architectures that work with the ( , ) labels, the following additional metrics are defined: From Fig. 6, Fig. 7, and Fig. 8 global performance in terms of are summarized for all cases considered. From these plots, it is possible to draw important considerations on the importance of labeling strategy, reference frame, and training method.
In Fig. 6 the box plot are organized from top-down in groups of 5 by macro-categories based on the labeling strategy and then in clusters of 4 based on the IP method, ordered from top-down as CELM, CNN, HCELM, and HCELM 3 . It is possible to immediately appreciate that all the methods trained with the labeling strategy based on ( , ) outperform all others considered. This is especially remarkable for the CELM method, which is performing similarly to the other IP methods when considering this labeling strategy. On the other hand, in all other cases, CELM-based methods perform poorly. This is especially true when compared with the CNN ones, which outperform all methods considered in all possible combinations. Focusing on the importance of the reference frame, it is observed that overall better performances are achieved when the labels are expressed in the reference frame than in the one, while the choice of the coordinate system (cartesian or polar) seems to cause only minor differences.
An even more concise representation is visible in the semi-log plot of the mean error clustered by body in Fig. 7 Fig. 7 it is also possible to observe a trend depending on the shape considered: simpler, regular shapes such as the one of Didymos (D) are better exploited for navigation than highly irregular ones such as Lutetia (L), 67P (P), and Hartley (H).
Finally, Fig. 8 reports in a stacked histogram plot the share for each dataset in which an IP method performs better than all others considered. While the CNN is always the best candidate, HCELM consistently scores as the second best method across all datasets considered. Moreover, the CELM is considered the best third option only in the case of the ( , ) labeling strategy. This graphic representation is possible thanks to the fact that the geometric points considered across the test sets are the same.
Since it is observed that the labeling strategy based on ( , ) works better than the others, the performance is further investigated in this case. In Fig. 9 the histograms of the and metrics in S 2 space are illustrated. When considering D, it is commented that the CELM shows a much larger mean error in than the other methods while only a smaller variability both in terms of means and variance is observed in . The fundamental failure mechanism for which the CELM method performs worse than the CNN one with D is thus represented by a larger mean error in the estimated CoF coordinates. When considering L and P, the difference between the histograms of the various methods is more subtle since only small variations in the mean and variance are observed across the different methods. On the other hand, when considering H, it is possible to see that CNN is much more accurate both in the CoF and range estimates than all other methods. The hybrid CELM performs better in the range estimate (with similar variance than the CNN) but does not perform at the same level as the CNN in the CoF estimate.
In Fig. 10 the error ellipses of the CoF coordinate in S 2 space are illustrated together with the error ellipse (dashed dark ellipse) that would have been obtained by not correcting the CoB with a data-driven scattering law implemented by the IP. It is also observed that different than all other ellipses, the CELM ellipse with D is not centered in the zero error point thus introducing a bias that is causing the larger mean error already observed in Fig. 9.
In Fig. 11 a representative case of the position error of the CELM in reference frame for the L body is illustrated.
It is possible to see that the position error is one order of magnitude higher in the boresight direction of the camera than on the other axis. This result is expected from optical-based navigation systems. This means that the error in the range estimate from the body is the major contributor to the positioning performances when using this labeling strategy.
Eventually, a visualization of a sample of 150 images in S 2 and the CoF prediction by the different IP methods is illustrated in Fig. 12.

IV. Conclusions
In this work, CELMs are investigated as a possible alternative to CNNs for autonomous vision-based navigation systems around small bodies. This is done in an extensive analysis considering 4 different small body shapes and 5 different labeling strategies, for a total of 20 scenarios, each of which is examined with 4 different IP methods, resulting in tens of thousands of different architectures explored.
It is demonstrated that the coupling between IP method and labeling strategy plays a fundamental role to achieve the desired performance. In particular, the ( , ) labeling strategy is found to be the best one. In this case, the navigation labels are given by two geometrical quantities which are estimated directly from the image, and a third, more difficult, label associated with the range from the body. The error on such a label is the one driving the error in the position estimate.
It is also noted that amongst the other labeling strategies, the ones adopting the reference frame perform better than those using the one. This hints at the fact that in the former the estimated position relies on the correlation with global geometric properties of the body which are not dependent on the specific rotational state but are rather dependent on the illumination conditions. On the contrary, in the latter, the methods need to learn illumination invariant features that correlate to the specific rotational state of the body. This seems to be much harder to do, at least when 7500 images are considered for training. Future work will be focused on changes in the method to allow larger datasets to be used. Irrespective of the reference frame, it is commented that the mapping from images to polar or cartesian coordinates seems to degrade the performance. In particular, while the ( , ) and ( 1 , 2 , ) labeling strategies both share as a label, the latter performs considerably worse than the former. This may hint at a more difficult mapping from images into two angles in spherical coordinates than the mapping to projective quantities in the image plane, the latter correlating much easily and directly from image.
Although the CNN outperform all other methods considered, the CELM seems a credible alternative, since it performs in the same order of magnitude as the CNN method when considering the best labeling strategy.
It is also commented that the usage of CELM for exploration of the architecture space generated CNN architectures that experienced robust training. This bootstrap training strategy could be deployed as a standard and efficient way to explore the global architecture space as suggested in [6]. Moreover, consistently with the findings in [4][5][6][7], a preference is observed for orthogonalization of the weights and biases of the kernels, as it is possible to see from the best hyper-parameters in Tab. 4.
Future works will address the addition of dropout into the networks to generate uncertainty estimates, algorithmic design choices to adapt the CELM for small body images, larger datasets, and the usage of segmentation maps as input in place of grayscale images, the former representing easier input to work with.


extfieldsofstudy:
Computer Science
Engineering

text:
Canonical Cortical Graph Neural Networks and its Application for Speech Enhancement in Future Audio-Visual Hearing Aids

Despite the recent success of machine learning algorithms, most models still face several drawbacks when considering more complex tasks requiring interaction between different sources, such as multimodal input data and logical time sequence. On the other hand, the biological brain is highly sharpened in this sense, empowered to automatically manage and integrate such a stream of information through millions of years of evolution. In this context, this paper finds inspiration on recent discoveries on cortical circuits on the brain to propose a more biologically plausible self-supervised machine learning approach that combines multimodal information using intra-layer modulations together with Canonical Correlation Analysis, and a memory mechanism to keep track of temporal data, the so-called Canonical Cortical Graph Neural networks. The approach outperformed recent state-of-the-art results considering clean audio reconstruction and energy efficiency, described by a reduced and smother neuron firing rate distribution, suggesting the model as a suitable approach for speech enhancement in audio-visual hearing aid devices.

Introduction
According to the World Health Organization (WHO), the number of people requiring hearing rehabilitation in the world is estimated to rise from 430 million ⋆ The authors are grateful to FAPESP grants #2013/07375-0, #2014/12236-1, #2017/02286-0, #2018/21934-5, #2019/07665-4, and #2019/18287-0, CNPq grants #307066/2017-7, and #427968/2018-6, as well as the Engineering and Physical Sciences Research Council (EPSRC) grant EP/T021063/1. nowadays up to 700 million until 2050, with nearly 2.5 billion people presenting at least some degree of hearing impairment [23]. Despite the impairment itself, deafness also impacts on social relationships [19,16] and environment perception [22], leading to other psychological and health conditions [15]. Over such circumstances, employing high-end energy-efficient technological approaches to build cross-modal sensory devices, i.e., combining audio and visual inputs to enhance hearing aid devices, seems a plausible way to improve individuals' life quality.
In the last decades, machine learning-based techniques have shown themselves as a suitable approach to tackle issues related to virtually any field of science, industry, or even daily life, ranging from computer vision [36] to medicine [29], and satellite imagery processing [12,28]. Machine learning also has been successfully employed in the context of speech enhancement [34,35], whose aim is to enhance speech quality and intelligibility when noise degrades them significantly [5]. Nevertheless, such methods may suffer massive performance degradation in the presence of overwhelming noise [7]. Many works address this problem using multimodal audio-visual (AV) information fusion. Combining AV information usually demands more sophisticated approaches, which intrinsically comprises several challenges, like data alignment, finding semantic gaps between low-level features and high-level information [8], and learning coherent and correlated latent patterns on different input modalities.
Combining noisy audio and clean images for clean signal reconstruction is analogous to reading the lips and body movements of a speaker in a boisterous environment, e.g., a pub with loud music, to obtain some additional information and create a context to enhance the information quality of a speech suppressed by the loud sound. Ngiam et at. [21], for instance, proposed a multimodal method capable of improving a target modality feature representation. Further, Adeel et al. [2] provided several improvements to the field, presenting a chaotic model for lip-reading integrating Internet of Things (IoT) and 5G Cloud-Radio Access Network, further improving AV information transmission for real-time speech reconstruction. Further work employ deep learning-based approach to exploits AV cues to estimate clean audio [4,20], also considering distinct language speakers [13].
Recently, Passos et al. [25] proposed a multimodal self-supervised Graph Neural Network (GNN) that combines AV data through using Canonical Correlation Analysis Graph Neural Networks (CCA-GNN) [37], also modeling the temporal information in the graph using the so-called prior-frame positional encoding. The method obtained outstanding results considering audio reconstruction and energy efficiency, analyzed in terms of neuronal activation rate.
Despite the advantages presented [25], the model lacks some points in the context of a biologically plausible approach. In this context, Passos et al. [24] proposed a multimodal approach using burst-dependent learning [26], a method inspired by more recent studies on the physiological mechanism of pyramidal neurons that regulates the learning by the frequency of bursts, where the credit assignment problem is addressed by the primary principles of pyramidal neurons suggested by Körding and König [18]. In parallel, the study of canonical cortical circuits [9,14] provides some interesting insights regarding the brain procedure toward multimodal information processing. Canonical cortical circuits model pyramidal neurons to receive different modalities of information, modulated in an excitatory or inhibitory fashion on deeper layers. Moreover, biologically plausible models should not underestimate the importance of memory in the learning process, which performs a fundamental role in information inference, acting as an intrinsic context for novelty comprehension.
The attributes mentioned above motivated the development of the Canonical Cortical Graph Neural Network, a novel self-supervised architecture that remodels and improves the ideas developed in [25] by introducing a more biologically plausible approach to modulating and filtering the multimodal information inside the so-called cortical graph layers. It also introduces a memory concept inside each cortical graph block, composed of a mechanism to "forget" irrelevant facts and update itself considering every consecutive node, which is presented in a logical time-step sequence using prior-frame-based positional encoding. Experiments conducted over the AV ChiME3 [5] dataset show the Canonical Cortical Graph Neural Network obtained state-of-the-art results, outperforming CCA-GNN in the contexts of faster convergence, audio reconstruction, and firing neuron activation rates. Regarding the latter, the model not only obtained lower rates of activation, but also produced smothier firing-rate distributions.
The main contributions of this paper are presented as follows: 1. To propose the Canonical Cortical Graph Neural Network, a biologically plausible model for multimodal information feature extraction. 2. To introduce a novel paradigm for intra-layer information fusion and memory modeling. 3. To provide a self-supervised energy-efficient model for correlated feature extraction for AV-based clean audio data reconstruction. 4. To foster the literature regarding speech enhancement and AV hearing aids.
The remainder of this paper is presented as follows. Section 2 provides a theoretical background regarding Graph Neural Networks with Canonical Correlation Analysis and the prior frame-based graph positional encoding, while Section 3 introduces the novel Canonical Cortical Graph Neural Network. Further, Sections 4 and 5 present the methodology and the experimental results, respectively. Finally, Section 6 states conclusions and future works.

Theoretical Background
This section provides a brief theoretical background regarding Graph Neural Networks with Canonical Correlation Analysis and the prior frame-based graph positional enconding.

Graph Neural Networks with Canonical Correlation Analysis
Let G = (X, A) be a graph where A ∈ R N ×N represents the adjacency matrix and X ∈ R N ×F the input data represented by graph nodes. Additionally, F represents the feature space dimension, and N denotes the number of nodes. The CCA-GNN [37] comprises three main steps, i.e., a random graph generator T , a graph neural network encoder f θ , where θ denotes the network's learnable weights, and an objective function based on Canonical Correlation Analysis. The graph generator produces two augmented versions of the same graph, which are presented to the graph neural network encoder for further computing and maximizing the canonical correlation between their outputs.
The idea behind such an approach is discarding decorrelated components while preserving correlated ones. In a nutshell, the model tries to keep the more significant information present in both augmented versions and to avoid individual behaviors, such as anomalies and noise. Figure 1 depicts the Canonical Correlation Analysis Graph Neural Network. Each sample represents a node in a graph whose edges describe the relationship between pairs of samples. The random graph generator T produces two augmented versions of this graph, which are employed to feed the GNN model. The output of both versions is compared using the canonical correlation analysis, and the network parameters are adjusted to maximize this metric.
The graph augmentation process comprises the same approach presented in [38,30], which conducts a random feature masking and edge dropping. In this context, each t i ∼ T comprises a distinct view, i.e., a transformed version of G, sampled at each iteration i.
The encoder is composed of a two-layered GNN but can be easily replaceable by any fancier model. The target function is designed to model the learning process as a canonical correlation maximization problem [10] using a self-supervised approach that considers two normalized views, Z A and Z B , produced over randomly augmented versions of the original graph. The objective is to maximize the correlation between these views, formalized as follows: where λ is a non-negative trading-off hyperparameter, and I is the identity matrix. The left term indicates the invariance term, which is responsible for minimizing the invariance between the two views. In contrast, the term on the right side describes the decorrelation term, which facilitates distinct features to capture different semantics through a regularization procedure. The terms in Equation 1 can be decomposed using a variance-covariance perspective [31]. Let s be an augmented version of the graph sampled from an input x, and Z s is a view of s obtained from a decoder output. The invariance term is minimized using expectation, described as follows: where V denotes the variance. Similarly, one can formalize the decorrelation term as follows: where ρ denotes the Pearson correlation coefficient, and Cov stands for the covariance matrix.

Prior Frame-based Graph Positional Enconding
A time-sequence based approach for graph positional encoding was recently proposed by Passos et al. [25]. The method computes the node neighborhood by connecting them to their k prior frame nodes in time sequence and attributing connection weights to the edges according to their distances in this time-based space. Figure 2 depicts this idea. The calculation of the edge weight w ij that connects a node i to a node j is performed as follows: where d ij denotes the distance from node i to node j in a frame-step space. Those values are stored in a distance matrix used to compute the positional encoding of the nodes.

Canonical Cortical Graph Neural Network
This paper presents a novel self-supervised approach for training multi-modal graph neural networks in a more biologically plausible way. In this context, the architecture combines several concepts inspired on cortical circuits observed in the brain [14,9] to model memory and multimodal information fusion with canonical correlation analysis [10], and to maximize the correlation of the information extracted from different inputs. Figure 3 presents a general overview of the proposed model, which suggests the more interesting procedures of the model are implemented at layer level. Therefore, Figure 4 illustrates this in-layer process, also depicting the behavior of each operation. A formal description of the procedures depicted in Figure 4 and performed inside each cortical layer is provided bellow. Firstly, one should compute the audio f a , visual f v , memory f m , and modulation f w filters as follow: and where σ stands for the Sigmoid function, W a , W v , W m , and W w , are the weight matrices for the audio, visual, memory, and modulation filters, respectively, and b a , b v , b m , and b w are the biases for the audio, visual, memory, and modulation filters, respectively, while [h a , h v ] denotes the concatenation of the audio h a and the visual h v graph convolution outputs. Further, the pre-modulation ρ is computed as follows: where W ρ and b ρ are the pre-modulation weight matrix and bias, respectively. Finally, the modulation ω is computed as follows: where ⊗ stands for the dot product. The memory µ is the more tricky updating since it considers both "forgetting" irrelevant memories using the memory filter f m and introducing new experiences presented in the modulation ω. Moreover, the operation is performed individually for each node n ∈ {0, . . . , N } since they are connected in a logi-cal temporal sequence established by the prior frame positional encoding [25], described as follows: where µ n , ω n , and f n m are the node's n memory, modulation, and memory filter. In the sequence, the memory is updated as follows: where W µ and b µ are the memory weight matrix and bias, respectively. Finally, the layer output, i.e., the new node representation of the audio and visual graphs h ′ a and h ′ v , respectively, are computed as follows: and Finally, h ′ a and h ′ v become the the node representations of the subsequent layer audio and visual graphs, respectively, in case of an intermediate layer. Regarding the output layer, the model performs the canonical correlation analysis between h ′ a and h ′ v using Equation 1 and backpropagates this value to optimize the network parameters.

Methodology
This section describes the dataset and configuration employed during the experiments.

AV ChiME3 Dataset
The dataset used in this paper aims to combine environmental information, i.e., audio and visuals, to train an efficient model for enhancing and amplifying clean audio signals considering multimodal data. The dataset comprises triples composed of image, clean audio, and noisy audio signals, from which the image and noisy audio denote the model's input while the clean signal stands for the desired output, i.e., the instance target in the context of supervised learning. The videos are extracted from Grid [11] dataset, in which different classes of noises (public transport, pedestrian area, street junction, cafe) with signal to noise ratios (SNR) ranging from -12 to 12dB extracted from ChiME3 [6] are introduced, composing the AV ChiME3 [5] dataset. Further, the samples are preprocessed to improve the sentence alignment and incorporate multiple visual frames to include temporal data. In total, the dataset contains 989 sequences from 5 different speakers, described as one black male, two white males, and two white females. Each sequence comprises 48 frames, summing up to a total of 47, 472 synchronized triples of samples.
Audio feature extraction Log-FB vectors were employed to extract both clean and noisy audio features. The technique samples the audio signal at 22, 050kHz for further segmenting it into M 16ms frames with 800 samples per frame and 62.5% increment rate. Furthermore, it uses a hamming window and Fourier transformations to produce a 2048-bin power spectrum. Moreover, it employs a logarithmic compression to obtain the 22-dimensional log-FB signals.
Visual feature extraction The samples extracted to generate the visual set of features were obtained using an encoder-decoder architecture over the Grid Corpus dataset. Lip-regions were detected using Viola-Jones [32] algorithm, for further tracking the frame sequence using the method proposed in [27]. Additionally, a manual effort was employed to randomly inspect the sentences, ensuring good lip tracking [1]. The encoder-decoder approach is then used to create vectors of pixel intensities, in which the 50 first components are vectorized in a zigzag order and then interpolated to match the equivalent audio sequence.

Experimental Setup
The experiments conducted in this paper compare the proposed Canonical Cortical Graph Neural Networks against the recent state-of-the-art model CCA-GNN for the task of multimodal clean audio reconstruction based on noisy audio and visual features, extracted using logFB and an encoder-decoder approach, respectively, as described in Section 4.1. Both models comprise a similar architecture composed of two hidden layers, the first comprising 512 and the second 256 neurons. The parameters were selected empirically. The learning is conducted to maximize the canonical correlation analysis for coherent feature extraction during 200 epochs using the Adam optimizer with a learning rate of 10 −3 and a trading-off parameter of λ = 0.0001 (Equation 1). At the same time, the hyperparameters of the CCA-GNN follow the configuration employed in [25]. Finally, the graphs are generated considering eight distinct prior-frame scenarios, i.e., k ∈ [3,5,7,10,15,20,25,30]. Notice that the plots provided in experimental results comprise only k ∈ [3, 10, 30] to illustrate the difference between a low, medium, and a high number of neighbors/prior frames.
After maximizing the canonical correlation analysis between the two channels, the features extracted by the networks are employed to feed a dense layer responsible for reconstructing the clean signal by minimizing the mean squared error as the cost function. The model is optimized during 2, 000 epochs using the Adam optimizer with a learning rate of 0.005 and a weight decay of 0.0004.
The dataset was divided into 20 folds to provide an in-depth statistical analysis. Each fold comprises 50 sequences of 48 frames each, summing up to 2, 400 samples per fold. As stated in Section 4.1, the dataset is formed by three subsets: (i) clean audio, (ii) noisy audio, and (iii) clean visual. The noisy audio and the clean visual input are used to feed the multimodal GNNs, while the clean audio is considered the reconstruction target. Finally, each fold is split into train, validation, and test sets, following the proportions of 60%, 20%, and 20%, respectively. The Wilcoxon signed-rank test [33] with 5% of significance was considered for statistical evaluation.

Experiments
This section exploits the superiority of the proposed Canonical Cortical Graph Neural Networks over the state-of-the-art CCA-GNN. The experimental results consider the contexts of feature extraction analysis, clean audio signal reconstruction, as well as neuronal activation and energy efficiency.

Feature Extraction Analysis
This section explores the task of self-supervised feature extraction in terms of canonical correlation analysis. The idea consists of extracting correlated features considering both the audio and visual channels, contributing to better features for clean audio reconstruction. In this context, the Canonical Cortical Graph Neural Networks, presented in Figure 5 as Cortical, showed a performance 75% higher than the CCA-GNN, on average, considering a small, medium, and high neighborhood, i.e., k ∈ [3, 10, 30].  Further, Table 1 provides the final MSE values over the testing set considering eight distinct k scenarios, i.e., k ∈ [3,5,7,10,15,20,25,30]. Notice bold values stand for the best values considering the Wilcoxon signed-rank test [33] with a significance of 5%. Such results show that the proposed model presents a better behavior when exposed to a reduced number of neighbors, i.e., the historical information is reduced, obtaining the lowest MSE overall over this scenario. This result can be explained by the memory implemented in the architecture, i.e., since the memory tries to model the predictions based on past frames, a longer temporal sequence makes this information ambiguous, leading the model to an extra exposure to past instances, thus overfitting. The idea is reinforced by the CCA-GNN approach, where an opposite behavior is observed in most cases, i.e., since CCA-GNN does not implement a memory, higher numbers of neighbors usually lead to lower prediction errors.

Clean Signal Reconstruction
Finally, Figure 7 depicts some examples of clean audio reconstruction for both models. Notice that the Canonical Cortical Graph Neural Networks with a reduced number of past frames, i.e., k = 3, obtained almost perfect reconstructions of the clean signal in both cases, which reinforces the idea that the memory implementation replaces the necessity of longer temporal information, described by a higher number of neighbors. Figure 7(b) shows that CCA-GNN is also capable of producing good representations when then the number of past frames is big enough, i.e., k = 30, even though the reconstruction is not as good as the proposed method.   Table 2 provides a comparison of the Canonical Cortical Graph Neural Network against the state-of-the-art results reported in recent works using the AV ChiME3 dataset and multimodal approaches for audio-visual speech enhancement, i.e., CCA-GNN and CCA Multilayer Perceptron (CCA-MLP) [25], a Long-Short Term Memory (LSTM) [3] and a Multilayer Perceptron (MLP) [5] based approaches, as well as a canonical correlation-based short-time objective intelligibility deep learning (CC-STOI DL) [17] method. Notice the proposed approach provided the most accurate results, outperformed all the compared tecniques.

Neuronal Activation Analysis
The neuronal activation rate analysis is fundamental for future audio-visual hearing aids since the metric reflects the model's energy efficiency, a critical feature considering energy-constrained environments like hearing aids and embedded devices. In this context, reducing the firing rate directly implies a reduction in energy consumption. Figure 8 depicts the neuron activation rate over the intermediate block layers concerning the audio and visual channels. Notice that both  A numerical representation is presented in Table 3, which comprises a more complete set of neighbors (past frames), i.e., k = [3,5,7,10,15,20,25]. The results reinforce the idea that the proposed model relies almost equally upon both noisy audio and visual modalities for the final output since their areas under the curve are practically the same for all cases, except for k = 10, with an irrelevant difference, i.e., 256 and 255 for noisy audio and visual, respectively. The results also show that the proposed model outperforms the baseline in the context of neuronal activation and, consequently, energy consumption in every possible scenario, showing itself as a suitable approach for speech enhancement in future audio-visual hearing aid devices considering both accuracy and energy performance.

Conclusion
This paper proposed a novel self-supervised method for multimodal correlated feature extraction through canonical correlation analysis maximization. The proposed model comprises a block-based neural network, where each block comprises two Graph Neural Networks layers, i.e., one for noisy audio and the other for the visual features, a memory, and a set of operations to filter, insert, delete, and modulate the input signals. Such operations are inspired in recent discoveries related to cortical cells and their interactions. Experiments were conducted over the AV ChiME3 dataset, designed for the task of multimodal clean audio reconstruction considering noisy audio and clean visual instances, compared the proposed approach against the CCA-GNN, a similar state-of-the-art method proposed recently for the task. Results show that the proposed Canonical Cortical GNN provides more coherent and better-quality features, reaching higher values of canonical correlation analysis. The proposed approach also obtained more accurate reconstructions, generating cleaner reconstructions. Moreover, the model delivers higher efficiency in terms of energy, evaluated by neurons' firing rate. Finally, it also showed itself to be less dependent on a more extended prior-frame sequence, i.e., high values of k, since the memory can store and track temporal information.
Regarding future work, we aim to extend a similar cortical-based architecture to Convolutional Neural Networks and applications to two-dimensional data. We also aim to implement the model on chips for training and inference for possible future implementation in hearing aid devices for AV speech enhancement.