NeDSeM: Neutrosophy Domain-Based Segmentation Method for Malignant Melanoma Images

Skin lesion segmentation is the first and indispensable step of malignant melanoma recognition and diagnosis. At present, most of the existing skin lesions segmentation techniques often used traditional methods like optimum thresholding, etc., and deep learning methods like U-net, etc. However, the edges of skin lesions in malignant melanoma images are gradually changed in color, and this change is nonlinear. The existing methods can not effectively distinguish banded edges between lesion areas and healthy skin areas well. Aiming at the uncertainty and fuzziness of banded edges, the neutrosophic set theory is used in this paper which is better than fuzzy theory to deal with banded edge segmentation. Therefore, we proposed a neutrosophy domain-based segmentation method that contains six steps. Firstly, an image is converted into three channels and the pixel matrix of each channel is obtained. Secondly, the pixel matrixes are converted into Neutrosophic Set domain by using the neutrosophic set conversion method to express the uncertainty and fuzziness of banded edges of malignant melanoma images. Thirdly, a new Neutrosophic Entropy model is proposed to combine the three memberships according to some rules by using the transformations in the neutrosophic space to comprehensively express three memberships and highlight the banded edges of the images. Fourthly, the feature augment method is established by the difference of three components. Fifthly, the dilation is used on the neutrosophic entropy matrixes to fill in the noise region. Finally, the image that is represented by transformed matrix is segmented by the Hierarchical Gaussian Mixture Model clustering method to obtain the banded edge of the image. Qualitative and quantitative experiments are performed on malignant melanoma image dataset to evaluate the performance of the NeDSeM method. Compared with some state-of-the-art methods, our method has achieved good results in terms of performance and accuracy.

Introduction
Malignant melanoma is a dangerous skin cancer, with a substantial death rate [1]. Skin lesion segmentation is the first and indispensable step in the the identification and diagnosis of malignant melanoma [2]. Due to the features shown in skin lesions and the skin itself among different people are diverse, the segmentation work is complicated. Meanwhile, the edges of malignant melanoma are irregular or have notches, etc., and the malignant melanoma does not have a smooth round or oval outline like ordinary nevus. The definition of skin lesion borders is also a difficulty [3][4][5]. Malignant melanoma can occur in any part of the body. It may occur on the skin surface of the abdomen and lower limb skin of the body. It can also appear in areas that do not heal easily, such as the lips, eyes, toenails, and on the surface of cavities such as the nose [6]. Experienced specialists are required when the malignant tumor resections are performed. Meanwhile, if the surgical injury is too large, the injury will not heal easily. If the wound is small, the skin lesion area architectures to segment the demarcation line/ridge and vessel automatically, namely, squeeze and excitation U-Net (SE U-Net) and U-Net with attention gates (AG U-Net). In basic U-Net, each channel of feature maps is weighted equally, whereas, in SE U-Net, the weights of each channel are adjusted adaptively by adding parameters to it. The AG U-Net selects only salient features of an image and prunes the irrelevant features by applying a weighted feature map function for each object and paying attention to a particular object in an image [21]. The novelty of the literature lies in detailed analysis and discussion of U-Net++ results and implementation of U-Net++ in lung segmentation using X-ray. A thorough comparison of U-Net++ with three other benchmark segmentation architectures and segmentation in diagnosing TB or other pulmonary lung diseases is also made in the literature [22]. A Full convolutional Residual Network (FcRN) for end-to-end training is proposed, and it obtains a better result of segmentation [23]. U-Net and Deeplabv3 are combined to built DeepLabv3+, a decoder module to recover the boundaries of the object is added as an extension of Deeplabv3. Recently, attention-based networks have been diffusely used in various problems of computer vision [24]. Mu et al. proposed channel context and dual-domain attention based U-Net (CDU-Net) for segmentation of skin lesion attributes. The channel context feature fusion module (CCM) is used to further extract context information. dualdomain attention (DA) modules are further combined to improve the accuracy of segmentation result [25]. Jignesh G. et al. proposed a new convolutional neural network-based approach for skin lesion segmentation. Meanwhile, a novel multi-scale feature extraction module is proposed for extracting more discriminative features for dealing with the challenges related to complex skin lesions; this module is embedded in the U-Net, replacing the convolutional layers in the standard architecture. Further, two different attention mechanisms refine the feature extracted by the encoder and the post-upsampled features [26]. Nida N et al. proposed a deep learning method for automatic detection and segmentation of melanoma regions within the dermoscopic images for precise melanoma segmentation. The proposed method generates bounding boxes around multiple regions to precisely detect the affected regions using RetinaNet. Further, conditional random field (CRF) is applied to the detected regions for segmentation of the melanoma lesion [27]. However, many segmentation methods using convolutional neural networks often fail to extract lesion boundaries accurately. Meanwhile, the edges of the lesions in medical images are fuzzy and constantly oscillating. However, deep learning methods do not preserve edges very well. Meanwhile, deep learning methods always require a high cost of medical image collection and labeling, it is difficult to collect enough and balanced training samples for medical image segmentation.
Neutrosophic Set (NS) theory provides a new method for solving uncertainty problems. NS method has been used in image processing and has achieved good results [28]. In general, the perception of the fuzzy-based approaches is generalized by the neutrosophic set. It includes the fuzzy set and intuitionistic fuzzy set [29]. In the NS domain, Guo and Cheng introduced fuzzy c-means clustering, and the indeterminacy of the image is calculated by the entropy. The research put forward an image segmentation model, the model combines NS theory with level set theory. The uncertainty of the image is reduced by using a filter which is newly defined. The boundary of the object is automatically extracted by using the level set algorithm [30]. Cheng and Guo converted the image in NS domain, and segmented the image through the thresholding-based segmentation method. The three NS memberships, that is True (T), Indeterminacy (I), and False (F) are generated. Next, the indetermination is estimated by calculating the entropy in NS. The uncertainty of the set is reduced by applying the mean operation [31,32]. Sert and Alkan proposed a Chan Vese segmentation approach on the basis of NS, and it is used for detecting the edge [33]. Zhang et al. combined the neutrosophy theory with image segmentation, the image is mapped in the NS domain. The image is segmented by using a watershed method [34]. Shan et al. proposed an neutrosophic L-means (NLM) for segmenting images. The method is a clustering procedure on the basis of NS [35]. However, the three memberships (T, I, F) in the neutrosophic set are independent of each other. In order to simulate human decision-Entropy 2022, 24, 783 4 of 20 making in the process of image segmentation, a single-valued Neutrosophic Entropy (NE) is constructed through T, I, and F to effectively represent the fuzzy boundary features of skin lesion area, healthy skin area and banded edge area in the image. The proposed NeDSeM method further improves the accuracy of the uncertainty of things reflected by the neutrosophic entropy. In this way, it can improve the description of the fuzzy features of the banded edge of the skin lesion area and assist the doctor in effectively resecting the skin lesion area. All knowledge is obtained from the image by unsupervised learning without too much reliance on human experience, and the result is more objective.
Although the above methods have achieved better results, many methods are difficult to transfer to the real world. Compared with the fixed human body structure in other medical images, the distribution of skin lesions in skin diseases often does not have inherent laws. For the segmentation of malignant melanoma, to the best of our knowledge, the representation of images in the existing methods does not conform to the medical diagnosis rules, and their segmentation results retain less edge information. Neutrosophy domain-based segmentation method (NeDSeM) is proposed in this paper according to the background of malignant melanoma image segmentation. Aim of our work is automatically confirm the resection range of skin lesions and effectively segment the edges of skin lesions in malignant melanoma images. It can assist doctors to perform precise skin lesions resection. The proposed method is evaluated on real dataset. Experimental results show that the segmentation accuracy of our method exceeds baseline methods. Overall, the main contributions of our work are listed below.

•
In this paper, a segmentation method integrated with morphology and neutrosophy is proposed to segment the banded edges in malignant melanoma images well. In this method, the pixel matrixes of images are optimized at multiple levels. The method in this paper can quickly and accurately obtain better segmentation effect by clustering the final optimized pixel matrixes.

•
Due to the banded edges between the skin lesion areas and the healthy skin areas in malignant melanoma images have stronger uncertainty and fuzziness. Neutrosophic Set is used to solve this problem that is superior to a fuzzy set. The three memberships in the neutrosophic set can well highlight the banded edges of images. This paper incorporates the closing operation and the gradient of the morphology into the neutrosophic set conversion process to get these memberships.

•
A new neutrosophic entropy model is proposed to comprehensively express T, I, F and highlight the banded edges in the images. The neutrosophic entropy is explained from its spatial geometric meaning, and a new concept of the isentropic cylinder is proposed in the perspective of the intuitiveness and the fuzziness.
Section Methods provides an overview of the proposed segmentation method and discusses the detail of the model. Section Experiments presents the performance evaluation of the method for malignant melanoma image segmentation. Section Conclusions concludes the paper and discusses the possible future works.

Overview of the Method
In this paper, an automatic segmentation framework for malignant melanoma images is proposed, as shown in Figure 1. Firstly, an image is converted into three channels, that is R, G and B, and the pixel matrix of each channel is obtained. Secondly, the boundaries between the skin lesion areas and the healthy skin areas in malignant melanoma images are fuzzy and uncertain. Therefore, a malignant melanoma image can be subdivided into a skin lesion area, a healthy skin area and a banded edge area between the skin lesion area and the healthy skin area. The three memberships in the neutrosophic set can well highlight the banded edges. NS is more successful than a fuzzy set in solving uncertain situations, especially in the application of image edge detection and segmentation. Therefore, the pixel matrixes are converted from RGB domain into NS domain by using the neutrosophic set conversion method. The NS matrixes are obtained and denoted as (T R , I R , F R ), (T G , I G , F G ) and (T B , I B , F B ). In the process of conversion, due to the closing operation in the morphology can be used to eliminate the holes or the small points in the objects and restore the real pixels of the target images. Therefore, the closing operation is used in the conversion process of T and F. Due to the gradient in the morphology is applied to highlight the edge contours of the target objects, it can represent the degree of uncertainty of T, I and F. Therefore, the gradient is used in the conversion process of I. Thirdly, the colour of the edges between the skin lesion areas and the healthy skin areas in malignant melanoma images is gradual change. However, the three memberships in the neutrosophic set are independent of each other. Therefore, a new neutrosophic entropy model is proposed to combine the three memberships according to some rules by using the transformations in the neutrosophic space. It can comprehensively express T, I, F and highlight the banded edges of the objects. It can map the two-dimensional image pixels to the three-dimensional neutrosophic space one by one. The irregular and uneven pixels in the image can be regularized by isentropic cylinder. Meanwhile, it can also represent the pixel distribution characteristics of malignant melanoma dermoscopic images and facilitate the calculation of segmentation model. The neutrosophic entropy matrixes are obtained and denoted as E R , E G and E B . Fourthly, the edges of the skin lesion areas of malignant melanoma images are irregular and constantly changing. It is difficult to describe the edge features accurately by using only a single component. Therefore, a feature augment method is established by the difference of the three components. It can reduce the complexity of the model and the consumption of computing resources. The effective edge information of malignant melanoma images are retained as much as possible. The neutrosophic entropy matrixes after using the feature augment method are denoted as R b . Fifthly, due to the dilation operation can enlarge the binary areas according to the structuring element, the dilation is used on the neutrosophic entropy matrixes and the matrixes after dilation operation (DI matrixes) are obtained. Finally, the image that is represented by transformed matrix is segmented by the Hierarchical Gaussian Mixture Model clustering method to obtain the banded edge of the malignant melanoma image.

Morphology-Based Operations
In this section, the closing operation, the gradient and the dilation in morphology are introduced, respectively. The closing operation and gradient are used in the process of NS conversion. The holes or the small points in the object can be eliminated by using the closing operation in the morphology and the real pixels of the target image can be restored. So the closing operation is used in the conversion process of T and F. The gradient is used to highlight the edge contour of the target object and it can represent the degree of uncertainty, so the gradient is used in the conversion process of I. The dilation operation can enlarge the areas according to the structuring element, so the dilation is used before clustering.
Firstly, the pixel matrixes of R, G and B in the images are extracted. Morphology is a

Morphology-Based Operations
In this section, the closing operation, the gradient and the dilation in morphology are introduced, respectively. The closing operation and gradient are used in the process of NS conversion. The holes or the small points in the object can be eliminated by using the closing operation in the morphology and the real pixels of the target image can be restored. So the closing operation is used in the conversion process of T and F. The gradient is used to highlight the edge contour of the target object and it can represent the degree of uncertainty, Entropy 2022, 24, 783 6 of 20 so the gradient is used in the conversion process of I. The dilation operation can enlarge the areas according to the structuring element, so the dilation is used before clustering.
Firstly, the pixel matrixes of R, G and B in the images are extracted. Morphology is a nonlinear filtering method. The basic idea is to measure and extract corresponding shapes to achieve image analysis and object identification using structural elements with a certain morphology. Morphological operation is formed by the interaction of the object shape set and structural elements. It is not only can suppress noise, but also can detect the real edge greatly [36]. Thus, the use of morphology can not only effectively filter out noise but also retain the original detail information in the image, and it has a better edge detection effect. The morphology contains two basic operations: Erosion and Dilation [37]. Examples of corrosion and expansion processes are shown in Figure 2. For an image, the results of erosion and dilation operation can be obtained.
where the erosion and dilation operations are represented by and ⊕, respectively. The results of the erosion and dilation are M E and M D . The pixel matrix of the image is denoted as P. The structuring element is denoted as S. Compared with structuring element, the smaller areas are removed by erosion operation. On the contrary, the binary regions are enlarged in the light of structuring element by using dilation operation. Based on the above formulas, the closing operation is obtained as the follo ( ) where the closing operation is represented by •, and it denoted as MC. The pixe the image is denoted as P. The structuring element is denoted as S. The closing has two steps, step 1: the erosion operation is applied to filter the binary area; dilation operation. The holes to connect the whole binary region are elimina closing operation.
The intensity of pixels in the image can be enhanced through morphologic based on the difference between dilation and erosion. It is used to highlight the tour of the target object. The mathematical expression is as follows: Based on the above formulas, the closing operation is obtained as the following: where the closing operation is represented by •, and it denoted as M C . The pixel matrix of the image is denoted as P. The structuring element is denoted as S. The closing operation has two steps, step 1: the erosion operation is applied to filter the binary area; step 2: the dilation operation. The holes to connect the whole binary region are eliminated by the closing operation. The intensity of pixels in the image can be enhanced through morphological gradient based on the difference between dilation and erosion. It is used to highlight the edge contour of the target object. The mathematical expression is as follows: where the morphological gradient is denoted as M G . M D and M E represent the results of the dilation and erosion.

The Neutrosophic Set Domain Conversion
The boundaries between the skin lesion areas and the healthy skin areas in malignant melanoma images are fuzzy and uncertain. The three memberships in the neutrosophic set can well highlight the banded edges of the areas. In this section, the pixel matrixes are converted into the form of neutrosophic set by using the neutrosophic set conversion method. In the process of conversion, the closing operation and the gradient in the morphology are used. Then the fuzziness of image edge can be solved better.
Neutrosophic Set (NS) is a useful theory for computing fuzzy situations. In recent years, NS has made a initial attempt in the applications of image processing. NS is more successful than a fuzzy set (FS) in solving uncertain problems, especially in detecting the edge of image and segmenting image [38,39]. All image pixels are subdivided into T, I, and F memberships when the NS theory is applied in image segmentation.
The images are converted to the NS domain, and the images are processed in the NS field [40]. P(i, j) is the pixel matrixes in the original images, where i = 1, . . . , m, j = 1, . . . , n. P C is the pixel matrix set of P(i, j) in R, G and B. The conversion formula is the follows: where, P are the pixel matrixes of any channel in P C . P are converted intop by the closing operation in morphology, wherep (i, j) is the pixel matrixes ofp , the maximum and minimum values ofp arep max andp min , respectively. The sum of T and F is 1. The pixel matrixes of δ(i, j) are obtained by the morphological gradient conversion and absolute value calculation for P . δ max and δ min are the maximum and minimum values of δ, respectively. In summary, the neutrosophic matrixes of R, G and B are obtained, denoted as (T R , I R , F R ), (T G , I G , F G ) and (T B , I B , F B ).

The Neutrosophic Entropy Domain Conversion
Due to the color of the edge of malignant melanoma image is gradual, it cannot effectively define the boundary between the lesion area and the skin area. As shown in Figure 3, the banded edge is existed between the healthy skin area and the skin lesion area in the malignant melanoma image.
In Figure 3, there is a gradually fuzzy banded edge between the inner edge and the outer edge. When doctors often determine the cutting range of the skin lesion in the banded edge, there is no effective en-bloc resection due to human subjective factors. The colour of the edge between the skin lesion area and the healthy skin area in malignant melanoma image is gradual change. However, the three memberships in the neutrosophic set are independent of each other. Therefore, the neutrosophic entropy model is introduced to combine the three memberships according to some rules by using the transformations in the neutrosophic space. It can comprehensively express T, I, F and highlight the banded edges of the objects. In this section, a new neutrosophic entropy model is proposed from the perspective of the intuitiveness and the fuzziness of image lesion features. The neutrosophic entropy is explained from its spatial geometric meaning, which further improves the accuracy of the uncertainty of things reflected by the neutrosophic entropy. A new concept of the isentropic cylinder is proposed, and a new neutrosophic entropy model is constructed. The existing neutrosophic entropy method can describe the fuzzy decision-making process of doctors during diagnosis, it can help doctors determine the cutting position of the lesions during the operation. However, it only transforms for pixel values and does not consider the distribution of pixels in the dermatoscopic image. Therefore, the concept of isentropic cylinder is proposed. On the two-dimensional plane, the distribution of the same or similar pixels are irregular and uneven, so a new neutrosophic transformation method is proposed to map the points on the two-dimensional image to three-dimensional space of the neutrosophic one by one. It ensures that the original similarity relationship of pixels remains unchanged, so that these similar points fall on a cylinder determined by a fixed axis and radius. The results after image pixel mapping processing can be expressed explicitly, which can represent the pixel distribution characteristics of malignant melanoma dermoscopic images. Meanwhile, it is beneficial to the calculation of segmentation model. In summary, the neutrosophic matrixes of R, G and B are obtained, d IR, FR), (TG, IG, FG) and (TB, IB, FB).

The Neutrosophic Entropy Domain Conversion
Due to the color of the edge of malignant melanoma image is gradua fectively define the boundary between the lesion area and the skin area. As ure 3, the banded edge is existed between the healthy skin area and the skin the malignant melanoma image. In Figure 3, there is a gradually fuzzy banded edge between the inne outer edge. When doctors often determine the cutting range of the skin banded edge, there is no effective en-bloc resection due to human subjecti colour of the edge between the skin lesion area and the healthy skin are melanoma image is gradual change. However, the three memberships in th set are independent of each other. Therefore, the neutrosophic entropy m duced to combine the three memberships according to some rules by usin mations in the neutrosophic space. It can comprehensively express T, I, F The neutrosophic set contains both the membership information, namely t, f, and the unknown information, namely i. Therefore, the neutrosophic set should be composed of two parts, one is fuzziness, which can be measured by |t − f|, and the other is intuitiveness, which can be measured by |i − i c |.
For the convenience of comparison, all points are projected on plane ACDF, as shown in Figure 5. Finally, only the entropy between Q i and x* i is compared, and it has E(Q 4 However, it is impossible to compare point x i with point Q 1 and point Q 8 , because when x i moves to Q 1 or Q 8 , increase and decrease characteristics of its intuitiveness and fuzziness are opposite. It is impossible to intuitively judge which aspect is more important through geometric figures, so the entropy cannot be compared. To solve this problem, a cylindrical surface is made with the intersection line z = 0.5 t = f of plane z = 0.5 and plane EGBO as the axis and d as the radius, as shown in Figure 6. As point A approaches point B, the fuzziness increases and the intuitiveness decreases. We can assume that their interactions cancel each other out, so that the uncertainty information at this point is constant, that is, the entropy is constant, that is, E(A) = E(B), this arc is called an isentropic cylinder.    As can be seen from Figure 6, the points on the isentropic cylinder ha of equal entropy, and the entropy has an obvious decreasing trend with the    As can be seen from Figure 6, the points on the isentropic cylinder ha of equal entropy, and the entropy has an obvious decreasing trend with the Therefore, the neutrosophic entropy corresponding to this point can be mea As can be seen from Figure 6, the points on the isentropic cylinder have the feature of equal entropy, and the entropy has an obvious decreasing trend with the increase of d. Therefore, the neutrosophic entropy corresponding to this point can be measured according to the distance between any point in the cube and the straight line z = 0.5 t = f . A more reasonable and reliable entropy formula can be established based on this. Theorem 1 can be obtained after the research.
Theorem 1. Suppose X is a domain, X = {x 1 , x 2 , . . . , x n }. It is a non-empty set, and the element of it is denoted as is called as a Neutrosophic Set (NS) on X. Where T A (x), I A (x), and F A (x) are the membership degree, the hesitancy degree, and the non-membership degree of the element x in X belonging to set A. For "x∈X, and −0 ≤ T A (x) + I A (x) + F A (x) ≤ 3+. That is, the three functions are independent of each other. For the convenience of study, it is called α =< T, I, F> is a neutrosophic number. Where ] − 0,1 + [ is the non-standard unit interval relative to the closed interval [0,1]. If Then E is the entropy of the intuitionistic fuzzy set.
In summary, the neutrosophic entropy matrixes are obtained, denoted as E R , E G and E B . The results processed by neutrosophic entropy not only better solve the problem of fuzzy edges of malignant melanoma images, but also more consistent with the decisionmaking of human. The detailed derivation process and some properties of single-valued neutrosophic entropy can be found in Supplementary Materials File S1.

Segmentation Method
The edges of the skin lesion areas of malignant melanoma images are irregular and constantly changing. Therefore, the difficulty of edge feature extraction is that it is difficult to describe using only a single component accurately. However, much computations are required by using three components for description at the same time. In this paper, the edge feature of the malignant melanoma image is established by the difference among the three components to reduce the complexity of the model and the consumption of computing resources. At the same time, the effective edge information of the malignant melanoma image is retained as much as possible. The influence of the blue channel component on feature extraction is considered. It is necessary to eliminate the area with a higher blue component value in the red component. Therefore, the edge feature of the malignant melanoma image is extracted, denoted as: where E R is the neutrosophic entropy conversion result of channel R, and E B is the neutrosophic entropy conversion result of channel B. Due to the dilation operation can enlarge the binary areas according to the structuring element, the dilation is used on the neutrosophic entropy matrixes. DI matrixes are obtained by using Formula (2). The processed image is shown in Figure 7. The HGMM algorithm adds hierarchical processing on the basis of t ian Mixture Model, and performs K Gaussian mixture modeling again distributions [12]. That is, the Gaussian mixture model is used to replace distribution. Suppose the image P contains N pixels, P = (p1, …, pn), the function is defined to divide the image into K categories.
where, G(xi|Θjr) is the Gaussian distribution. Π = {πij, j = 1, 2, …, K, i parameter set, Θ = {μjr, Σjr, j = 1, 2, …, K, r = 1, 2, …, R} is a parameter set prior probability of the pixel xi belonging to the jth category, ηjr represen ability that this pixel point belongs to the rth Gaussian distribution in th In order to increase noise immunity, Markov random field is introd erarchical Gaussian mixture model. The joint probability density functio is known by Bayes theorem of Formula (18). Finally, the Hierarchical Gaussian Mixture Model clustering method is used to segment the skin lesion areas of malignant melanoma images.
The HGMM algorithm adds hierarchical processing on the basis of traditional Gaussian Mixture Model, and performs K Gaussian mixture modeling again with K Gaussian distributions [12]. That is, the Gaussian mixture model is used to replace the kjth Gaussian distribution. Suppose the image P contains N pixels, P = (p 1 , . . . , p n ), the following density function is defined to divide the image into K categories.
where, G(x i |Θ jr ) is the Gaussian distribution. Π = {π ij , j = 1, 2, . . . , K, i = 1, 2, . . . , N} is a parameter set, Θ = {µ jr , Σ jr , j = 1, 2, . . . , K, r = 1, 2, . . . , R} is a parameter set. π ij represents the prior probability of the pixel x i belonging to the jth category, η jr represents the prior probability that this pixel point belongs to the rth Gaussian distribution in the jth category. In order to increase noise immunity, Markov random field is introduced into the hierarchical Gaussian mixture model. The joint probability density function of Formula (19) is known by Bayes theorem of Formula (18).
where Π = {π ij , i = 1, 2, . . . , N, j = 1, 2, . . . , K} is parameter set, t is the iteration times, Z is the ormalized constant, the value of W is usually 1. g (t) ij is the probability information that the pixel x i belongs to the jth category combined with neighborhood information. β is the parameter of adjust the image smoothness. z mj in Formula (21) is the posterior probability. N i is the neighborhood information of pixel x i , the neighborhood is 3 × 3. N i is the index value set of the neighborhood point of the pixel x i . During the iteration process, update µ j , ∑ j , π ij . The formulas are as follows, Take the logarithm Likelihood function: The iterative estimation is performed for the Formula (25). When log (t+1) −log (t) < 0.1, then the iteration is stopped. The posterior probability function Z ij is maximized, and then the image segmentation is realized.

Experiments
In this section, the performance of the proposed malignant melanoma image segmentation method is evaluated. At first, the experimental environment and the used dataset are described. Then, the results of our experiments are discussed.

Dataset and Training Settings
The experiments are conducted on a computer equipped with an NVIDIA GTX 3090 GPU. The manufacturer of the GPU is Shenzhen Colorful Technology Development Limited Corporation, and its country of origin is America. The memorycard is Kingston FURY 128 GB (32 G × 4) DDR4 3200. The motherboard is Gigabyte X570 AORUS PRO WIFI. Our method is implemented in CUDA 10.0, cuDNN 7.5, tensorflow2.0, and python 3.7. The malignant melanoma images are obtained partly from a well-known local hospital and from ISIC2018. Because the collected dataset is small, it is difficult to effectively train the neural network model. In this paper, the training data is the training set of ISIC2018 dataset, and the models are tested on ISIC2018 dataset and our collected dataset.

Evaluation Metrics
For performance evaluation, we adopt several metrics recommended by the ISIC, namely, accuracy (Acc), specificity (SP), Jaccard index (JA), and Dice coefficient (Dice). The formulas for these evaluation criteria are: where the skin lesion pixel is assumed to be a positive class and the background pixel is a negative class. TP, FP, TN, and FN represent the numbers of true positives, false positives, true negatives, and false negatives, respectively. Given a set of images, Acc is the accuracy, SP is the specificity. JA computes the similarity between the predicted region of the object present in the image and the ground truth region, representing the ratio of overlap between the predicted segmentation results and the ground truth labels. Dice is a common performance metric to evaluate the segmentation results, which measures the gap between the segmentation results and the label.

The Visualized Results after Applying the Neutrosophic Entropy
In this paper, the results of R, G, B, and R b are transformed by neutrosophic entropy, and the comparison figure is obtained through the three-color scale processing of blue-yellow-red as shown in Figure 8. The R component can effectively extract the edge information of the skin lesion area. However, only the inner edge information of the skin lesion is captured for the area band with a gradually shallower edge. At the same time, the characteristic influence of the lesion area is effectively suppressed. The background contains a lot of noise information caused by hair and bubbles in the dermoscopic image. The G component captures the edge information on the outside of the image, which weakens the influence of the background noise compared with the R component, but the influence on the inside of the lesion area is more significant. The B component also captures the outer edge features of the edge while effectively suppressing the influence of background noise. However, it is also affected by the feature of the lesion area inside the skin lesion. R b combines the advantages between the two, and it can effectively divide the inner edge and the outer edge between the edge band and the blue-white veil background. At the same time, the influence of a large amount of background noise and internal features of skin lesions is suppressed. Finally, the morphology is used to remove a large amount of background noise, and at the same time, the small edge roughness is removed.
where the skin lesion pixel is assumed to be a positive class and the backgroun negative class. TP, FP, TN, and FN represent the numbers of true positives, fals true negatives, and false negatives, respectively. Given a set of images, Acc is th SP is the specificity. JA computes the similarity between the predicted region o present in the image and the ground truth region, representing the ratio of o tween the predicted segmentation results and the ground truth labels. Dice is performance metric to evaluate the segmentation results, which measures t tween the segmentation results and the label.

The Visualized Results after Applying the Neutrosophic Entropy
In this paper, the results of R, G, B, and Rb are transformed by neutrosoph and the comparison figure is obtained through the three-color scale processi yellow-red as shown in Figure 8. The R component can effectively extract the mation of the skin lesion area. However, only the inner edge information of the is captured for the area band with a gradually shallower edge. At the same tim acteristic influence of the lesion area is effectively suppressed. The background lot of noise information caused by hair and bubbles in the dermoscopic im component captures the edge information on the outside of the image, which w influence of the background noise compared with the R component, but the in the inside of the lesion area is more significant. The B component also capture edge features of the edge while effectively suppressing the influence of backgro However, it is also affected by the feature of the lesion area inside the skin lesio bines the advantages between the two, and it can effectively divide the inner ed outer edge between the edge band and the blue-white veil background. At the the influence of a large amount of background noise and internal features of s is suppressed. Finally, the morphology is used to remove a large amount of b noise, and at the same time, the small edge roughness is removed.

The Visualized Results of the Clustering Methods
The classic K-means, Fuzzy c-means (FCM), and Hierarchical Gaussia Model (HGMM) clustering methods are used to segment the images without m ical processing after the neutrosophic entropy processing, and the segmentat are compared. As shown in Figure 9, the first to fourth columns show the clus mentation results of the R, G, B, and Rb components, respectively. As the resul ment 3.2.1, the Rb component is superior to the single component in capturin

The Visualized Results of the Clustering Methods
The classic K-means, Fuzzy c-means (FCM), and Hierarchical Gaussian Mixture Model (HGMM) clustering methods are used to segment the images without morphological processing after the neutrosophic entropy processing, and the segmentation results are compared. As shown in Figure 9, the first to fourth columns show the clustering segmentation results of the R, G, B, and R b components, respectively. As the result of experiment 3.

The Visualized Results of the Morphology Method
The combination of neutrosophic entropy and morphological technology is used t improve the identification and representation of edge band features in the lesion area o malignant melanoma images. The problem of irregular edges, blurring and color grada tion of malignant melanoma images is well solved, and the skin lesion area segmentation performance is greatly improved. In the segmentation process of our method, some step can add morphological methods to fill in holes and missing points. Since morphology wil affect subsequent processing steps, the recall rate of image pixels is reduced. Therefore the later is morphology added, the better the original information of the image can b guaranteed. However, at the same time, the irregular and frizzy edges of the skin lesion area will affect the accuracy of the malignant melanoma edge segmentation. Therefore the results of adding morphological filling techniques in different links are compared t determine the best position of adding morphology. The visualization of the processin results after adding morphology in different steps is shown in Figure 10. It shows th transformation results after adding morphology of five malignant melanoma images in Rb, first expansion and then Rb, first Rb and then expansion.

The Visualized Results of the Morphology Method
The combination of neutrosophic entropy and morphological technology is used to improve the identification and representation of edge band features in the lesion area of malignant melanoma images. The problem of irregular edges, blurring and color gradation of malignant melanoma images is well solved, and the skin lesion area segmentation performance is greatly improved. In the segmentation process of our method, some steps can add morphological methods to fill in holes and missing points. Since morphology will affect subsequent processing steps, the recall rate of image pixels is reduced. Therefore, the later is morphology added, the better the original information of the image can be guaranteed. However, at the same time, the irregular and frizzy edges of the skin lesion area will affect the accuracy of the malignant melanoma edge segmentation. Therefore, the results of adding morphological filling techniques in different links are compared to determine the best position of adding morphology. The visualization of the processing results after adding morphology in different steps is shown in Figure 10. It shows the transformation results after adding morphology of five malignant melanoma images in R b , first expansion and then R b , first R b and then expansion.
It can be seen from Figure 10 that in the R b component of the raw image, there is still noise even if channel R is subtracted from channel B due to a large amount of original noise in the image, which affects the final segmentation effect. A large amount of noise can be effectively avoided either by first conducting morphology and then extracting the R b component or by extracting the R b component and then conducting morphology. However, it can better capture the edge band information of the skin lesion area by first extracting the R b component and then performing morphology for the internal features of the skin lesion area. At the same time, it has a better effect on edge burrs, edges, and corners, etc., as shown in the red circle in Figure 10. In summary, our method can improve segmentation accuracy through neutrosophic entropy conversion, and it can effectively remove noise by morphological methods. The improved image feature extraction method is helpful to capture irregular contour features effectively.  Figure 10. In summary, our method can segmentation accuracy through neutrosophic entropy conversion, and it can e remove noise by morphological methods. The improved image feature extractio is helpful to capture irregular contour features effectively.

Comparison with Other State-of-the-Art Segmentation Methods
The accurate segmentation technique for malignant melanoma proposed i per can be used in an unsupervised adaptive contour model. This technique hel arate the required banded edge entities from the background and skin lesions. T pixels are blurred by neutrosophic entropy, and then the information is colle neighboring pixels of a similar class. Thus the effective segmentation of malign noma of different intensities is ensured.
In order to prove the advantages of our method in fuzzy edge segmentati lignant melanoma, our method is compared with other state-of-the-art segm methods. Our proposed algorithm achieved the highest score in most of the per indicators used. In terms of evaluation indicators, Acc, SP, JA, and Dice are use provides a quantitative comparison between some state-of-the-art methods

Comparison with Other State-of-the-Art Segmentation Methods
The accurate segmentation technique for malignant melanoma proposed in this paper can be used in an unsupervised adaptive contour model. This technique helps to separate the required banded edge entities from the background and skin lesions. The image pixels are blurred by neutrosophic entropy, and then the information is collected from neighboring pixels of a similar class. Thus the effective segmentation of malignant melanoma of different intensities is ensured.
In order to prove the advantages of our method in fuzzy edge segmentation of malignant melanoma, our method is compared with other state-of-the-art segmentation methods. Our proposed algorithm achieved the highest score in most of the performance indicators used. In terms of evaluation indicators, Acc, SP, JA, and Dice are used. Table 1 provides a quantitative comparison between some state-of-the-art methods and our method. It can be seen from Table 1 that our method obtained the highest scores in Acc and SP, and it is only 0.002 worse than the optimal method in Dice. It shows that the segmentation result of our method is closer to the expected segmentation object. Meanwhile, compared with the deep learning method, our method greatly reduces the training time, and the results are relatively stable. Therefore, NeDSeM method in this paper has better segmentation performance than other advanced malignant melanoma segmentation methods.

Visualization of the Results
To further evaluate our proposed method and visually evaluate the segmentation results of different methods, three baseline methods with high segmentation accuracy in Table 1 are selected. The segmentation results of eight representative malignant melanoma images from our dataset and the ISIC2018 dataset are shown in Figure 11. In the visual comparison results, the comparison results of the edge region quality after image segmentation with the different methods are showed. Combining the sample information and analyzing the lesion segmentation in Figure 11, it can be found that the contour of the lesion can be roughly segmented using Deeplabv3+, and a large amount of edge information is not effectively segmented. The contour of the skin lesion area can be successfully segmented by using Jignesh's method. However, the result of segmentation still has the phenomenon of edge information loss. At the same time, many background skin pixels are inefficiently segmented, and the quality of the segmented edges is low. The edge information is effectively segmented and the segmentation result is smoother by using Nida's method. However, there are still large deviations between the edge shape and the expected in the case of edge fuzzy is serious, which causes many skin pixels to be misclassified. In the segmentation results of our method, the segmentation edges are smoother, and the shape of the outer edges is more similar to the expected segmentation, which effectively captures the irregular edge shape features of the skin lesion area of a malignant melanoma image. However, the morphological method is used for smoothing, resulting in the situation of red circle marking in our method, that is, the violent and similar local edge fluctuations are not effectively segmented. To further evaluate our proposed method and visually evaluate the segmentation results of different methods, three baseline methods with high segmentation accuracy in Table 1 are selected. The segmentation results of eight representative malignant melanoma images from our dataset and the ISIC2018 dataset are shown in Figure 11. In the visual comparison results, the comparison results of the edge region quality after image segmentation with the different methods are showed. Combining the sample information and analyzing the lesion segmentation in Figure 11, it can be found that the contour of the lesion can be roughly segmented using Deeplabv3+, and a large amount of edge information is not effectively segmented. The contour of the skin lesion area can be successfully segmented by using Jignesh's method. However, the result of segmentation still has the phenomenon of edge information loss. At the same time, many background skin pixels are inefficiently segmented, and the quality of the segmented edges is low. The edge information is effectively segmented and the segmentation result is smoother by using Nida's method. However, there are still large deviations between the edge shape and the expected in the case of edge fuzzy is serious, which causes many skin pixels to be misclassified. In the segmentation results of our method, the segmentation edges are smoother and the shape of the outer edges is more similar to the expected segmentation, which effectively captures the irregular edge shape features of the skin lesion area of a malignant melanoma image. However, the morphological method is used for smoothing, resulting in the situation of red circle marking in our method, that is, the violent and similar loca edge fluctuations are not effectively segmented.

Discussion
Supervised segmentation methods have proved to be extremely effective for many problems in medical image segmentation, but the performance of these methods heavily depends on the quality of the labels. However, in practical applications, labels are usually lacking or even no labels, and large-scale labeling on medical images will also cost a lot of labor and time costs. Therefore, most of the current methods use unsupervised learning methods. Especially in recent years, with the further research of deep learning, unsupervised deep learning methods have made great progress in the field of image segmentation. They have great potential for effective image segmentation by utilizing large amounts of unlabeled data.
For semantic segmentation, some networks like U-Net also handle feature map reduction and detail loss. When recovering the resolution to the input size, the feature maps in the top and shallow layers are concatenated, and details are directly used to restore object boundaries. Although this skip connection mechanism can improve segmentation performance, directly taking features from shallow layers induces noise in the top layers, thus undermining the segmentation accuracy. Meanwhile, U-Net is more suitable for segmenting larger images, such as contours of brain region. However, malignant melanoma dermatoscopic images usually do not have such a large scale. Compared with the fixed human body structures in other medical images, the distribution of skin lesions in skin diseases is not very regular, and the semantics are more complex. So it is difficult to provide target object recognition information for U-Net in low resolution. This is the reason why the performance of U-Net-based method is lower than other methods in Table 1. Therefore, the characteristics of the U-Net method itself lead to scenes that are not suitable for dermoscopy image segmentation, and this paper also proves this point from the aspect of experiment. Similarly, in Deeplabv3+, since malignant melanoma dermoscopic images have no fixed rules similar to natural images, the distribution of medical features in the images is mostly irregular, with fuzzy and irregular edges. Therefore, the knowledge obtained from training is only included in the training set after Deeplabv3+ performs encoder and decoder operations. However, some novel edge shapes or pixel distribution features are difficult to be matched effectively by the model in verification and actual use. This results in the segmentation result of Deeplabv3+ in Figure 11, which divides a large amount of normal skin into skin lesion area. In the literature [27], image features are extracted by FPN, and Conditional Random Field (CRF) is used for fine segmentation of images. CRF obtains as much information as possible from neighboring points and encourages similar pixels to be assigned the same label, while pixels with a large difference are assigned different labels. In this way, CRF can segment the image at the boundary as much as possible, so as to obtain more accurate segmentation results. But CRF only operates on adjacent nodes, which will lose some contextual information. In response to this problem, the difference of the fully connected CRF model is that it describes the relationship between each pixel and all other pixels. The target region can be greatly refined and segmented on the image by using this model, which is also the main reason why the method in reference [27] in experiment 3.2.4 has better performance on the two indicators of JA and Dice. It is confirmed that the accuracy of the method in the literature [27] on the dermoscopy image segmentation task is higher than that of the FCN method and the SegNet method. However, this method consumes more computing power and is more expensive to restore the local detailed structure of the target.
Deep learning-based methods can automatically learn advanced features from raw data, however, this extraction method is not limited by human rules. In medicine, the impact of this situation will be magnified, even sometimes deviat from the strict requirements of the medical field and wast valuable experience accumulated over the years in the medical field. Therefore, we prefer human in rules, so that the final result has a certain degree of interpretability. Compared with the above deep learning methods, we focus on finding a good space or representation method to characterize the spatial features of pixels in malignant melanoma images. The complex dermoscopy image segmentation problem is transformed into a clustering problem in the neutrosophic space. The neutrosophic conversion and neutrosophic entropy methods proposed in this paper can flexibly map complex pixel distributions to different isentropic cylinders, enhance the accurate representation of the fuzziness and intuitiveness of pixel categories, and measure the similarity of the entropy values of all pixels in the space to achieve image segmentation. Experimental results show that the proposed method has a significant improvement in segmentation accuracy, and our method is better than the baseline methods on two metrics (ACC is 0.9678, SP is 0.9813). The visualization is validated on the real malignant melanoma dermoscopy image dataset collected by us in a well-known local hospital and ISIC2018 dataset, and the effectiveness of our method is also demonstrated through Experiment 3.2.4 and Experiment 3.2.5. In this paper, the banded segmentation edge is more in line with the needs of doctors for diagnosis, that is, en-bloc resection of the skin lesion area is guaranteed for different body incidence locations, which shows the effectiveness and practicability of the method in this paper.

Conclusions
Starting from the real application scenario, this paper simulates the real diagnosis process of the doctor through a long-term close cooperation with a well-known local hospital. Therefore, a representation of the fuzziness and intuitiveness in the doctor's decision-making process is added into our proposed NeDSeM method. The fuzzy modeling of malignant melanoma images is realized by the neutrosophic conversion and neutrosophic entropy methods proposed in this paper. By segmenting the skin lesions of malignant melanoma images, it can assist doctors in the rapid and automatic preliminary diagnosis of dermoscopic images, unify the medical segmentation standards of dermoscopic images, reduce the influence of human bias, and reduce the heavy work pressure of doctors in diagnosing the skin lesion areas of a large number of dermatoscopic images. Our method mainly contains three modules, a NS domain conversion module, a NE domain conversion module, and an image segmentation module. In the NS domain conversion module, the gradient and closing operations in morphology are used to calculate indeterminacy (I) and true (T), respectively. The NS matrixes are obtained by the NS conversion method. The NE domain conversion module considers both the intuitiveness and the fuzziness of images and proposes a new neutrosophic entropy model to further improve the accuracy of the banded edges of the segmented images. The image segmentation module uses the morphological method to expand the NE matrixes, and the obtained DI matrixes are segmented by HGMM clustering method. The experimental evaluation of our malignant melanoma dataset and the ISIC2018 dataset demonstrates that our method has higher accuracy and better effect. In the future work, it will be verified that the NeDSeM method can be extended to other medical image segmentation tasks.


extfieldsofstudy:
Computer Science
Engineering

text:
VarArray: Array-Geometry-Agnostic Continuous Speech Separation

Continuous speech separation using a microphone array was shown to be promising in dealing with the speech overlap problem in natural conversation transcription. This paper proposes VarArray, an array-geometry-agnostic speech separation neural network model. The proposed model is applicable to any number of microphones without retraining while leveraging the nonlinear correlation between the input channels. The proposed method adapts different elements that were proposed before separately, including transform-average-concatenate, conformer speech separation, and inter-channel phase differences, and combines them in an efficient and cohesive way. Large-scale evaluation was performed with two real meeting transcription tasks by using a fully developed transcription system requiring no prior knowledge such as reference segmentations, which allowed us to measure the impact that the continuous speech separation system could have in realistic settings. The proposed model outperformed a previous approach to array-geometry-agnostic modeling for all of the geometry configurations considered, achieving asclite-based speaker-agnostic word error rates of 17.5% and 20.4% for the AMI development and evaluation sets, respectively, in the end-to-end setting using no ground-truth segmentations.

INTRODUCTION
The last decade has witnessed transformational progress in automatic speech recognition (ASR) technology driven by deep learning approaches. Modern ASR systems can accurately transcribe pre-segmented utterances recorded in acoustically moderate environments, as demonstrated in many ASR benchmarks such as Lib-riSpeech [1] and Switchboard [2], and are widely used in our daily lives. Yet, there are many challenges that must be overcome for the ASR technology to become usable more broadly [3,4]. One of the unsolved challenges is transcribing unsegmented natural conversations, where multiple people talk over each other in a spontaneous and thus unpredictable way.
Continuous speech separation (CSS) was proposed to handle the speech overlaps in natural human-to-human conversation transcription [5][6][7]. With this approach, continuously captured long-form conversational signals are split into multiple overlap-free signals. Since each of the output signal does not contain the speech overlaps, they can be processed with conventional ASR systems. A microphone array-based approach has been particularly successful and was applied to real meeting recordings [6] while most previous studies used artificially created meeting-like data.
This paper sheds light on the generalizability of the multi-channel speech separation models. Two major approaches exist for the multichannel neural network modeling for speech separation. One approach uses multi-channel features confined to a specific microphone array geometry [8,9]. While it can potentially utilize the full spatial information, the trained model can be used only for the microphone array that it is trained for. Another approach is to apply the same speech separation model to each input channel and merge the outputs from the individual channels by calculating their median or average [10,11]. While being applicable to any microphone array devices, this approach does not take advantage of the nonlinear correlation between the channels. Also, the computational cost becomes too large to deploy in practice when the number of channels is large. This problem is addressed by VarArray, our proposed array geometry-agnostic speech separation model. The input is a set of magnitude and inter-channel phase difference (IPD) features, organized in a way that is invariant to the microphone permutation. The model consists of conformer blocks [12,13] interleaved with transform-average-concatenate (TAC) layers [14] to exploit the spatio-temporal patterns exhibited in the input. TAC is a crosschannel layer model that can cope with any number of channels in a permutation-invariant fashion. To reduce the growth rate of the model evaluation cost with respect to the number of microphones, the multiple feature streams are merged early in the network.
Large-scale evaluation was carried out by using real meeting recordings. Two meeting corpora were used to evaluate the performance sensitivity to the array geometry: AMI [15] and Microsoft internal meetings [6]. Our meeting transcription system achieved state-of-the-art results in AMI under the condition of no groundtruth segmentations being used. Results of fine-tuning the speech separation model to AMI with an ASR-based loss function are also presented, demonstrating its impact on the full-blown system. Note that, while the array-geometry-agnostic modeling was studied before in related areas, such as dereverberation [16], noise reduction [17], and ad hoc microphone arrays [18], this paper is the first to combine different pieces including the IPD features, conformer speech separation, and TAC in a cohesive and efficient way and presents thorough evaluation results for the real meeting tasks.

General architecture
CSS is a front-end-based approach to the speech overlap problem in natural human-to-human conversation transcription. Given longform audio signals captured by a microphone array, the goal of CSS is to generate signals that have the same length as the input in such a way that each of the signals does not contain overlapped utterances inside while the sum of these signals retains the spoken content present in the input signals in its entirety. It is tempting to try to estimate the clean speech signal of each speaker, which means is equal to the total number of speakers. However, this approach is extremely difficult for streaming processing because it requires the front-end to perform speaker diarization in addition to enhancing the speech. A more practical approach, which is increasingly being adopted, is to address only the speech overlap problem [7,9,19,20]. In meetings, the maximum number of simultaneously talking speakers is very limited. In fact, we can assume only two or fewer speakers to be active for the majority of the meeting time [21]. This means that only two output signals are sufficient in practice (i.e., = 2). If the audio segment being processed has two overlapping utterances, we separate each utterance and send the obtained signals to different output channels. When the current segment consists of only one speaker, we just have to enhance the speech quality and route the processed signal to one of the output channels. The unused output channel can be filled by zero to make the two output signals synchronous. As the output signals are overlap-free, they can be directly fed to a conventional ASR system.
A sliding window-based approach is often adopted to enable CSS. At each window position, a speech separation model trained with utterance-level permutation invariant training (uPIT) [22] is applied to the windowed input signal to generate separated signals. The order of the output signals is determined to maintain the consistency with the separated signals obtained at the previous window position. Specifically, we choose the output permutation that provides the smallest mean squared error between the pairs of the separated signals obtained at the current and previous window positions. The errors are calculated for the overlapping frames of the two windows. This "stitching processing" can be done by using the overlapped segment between the two adjacent windows. In the following, we focus our attention on the local speech separation and uPIT as it constitutes the central piece of the sliding window-based approach.

Local speech separation
The local speech separation process can be described as follows. Let denote the short time Fourier transform (STFT) coefficient of the audio signal observed by the th microphone, where and represent the frequency and time indices, respectively. Feature vector is computed for each time frame, and the sequence of the feature vectors is fed to a speech separation neural network model. The model produces time-frequency (TF) masks for predicting the two clean speech signals (recall that we assume = 2) and the background noise. The masks are used to perform minimum variance distortionless response (MVDR) beamforming to estimate the clean signals. It was empirically found that sparcifying the TF masks before the MVDR computation improved the ASR accuracy. In our experiments, this was performed by retaining only the most dominant sound source for each TF bin. Gain adjustment of [8] was also applied after beamforming to counteract MVDR's inability of generating complete silence for non-overlapped signals.
During training, reference signals are available for the clean speech and the noise. For training samples with one active speaker, one of the reference signals is a zero sequence. The uPIT loss [22] is employed to cope with the arbitrariness of the order of the two speech signals.
The feature vector, , can be the concatenation of the STFT coefficients of all the microphones [9] or a stack of IPDs [8], just to mention a few examples. Various speech separation models can be applied atop these concatenated features, including conformer networks [13] and dual-path networks [23]. However, using the concatenated features make the system dependent on a particular microphone array and usable only with the audio device which it is trained for. Our goal is to create a speech separation model that can work with any microphone arrays which performs as well as or better than the array geometry-dependent ones.

VARARRAY
VarArray, our proposed speech separation model, can efficiently utilize the spatial and temporal information from any number of microphone inputs for accurate TF mask estimation. It also yields the same output regardless of the microphone permutation. There are two key ingredients: one concerns the feature design; the other relates to the model structure. They are described in Sections 3.1 and 3.2, respectively. In the following, M, F, and T denote the index sets of the microphones, frequency bins, and time frames, respectively.

Feature set
With VarArray, we use a variable-size feature set, { } ∈M , for each time frame instead of stacking all the features obtained from different microphones in a particular order. This prevents the input features from being tied to a specific microphone array with a predefined channel indexing system. The simplest way is to use the STFT coefficients of the th microphone as . Below, we propose a normalized IPD-based feature set, which outperformed the STFT-based method in our preliminary experiment.
With the proposed feature set, is defined as the concatenation of the magnitude of an average spectrum and the channelwise IPD with respect to the average spectrum. That is, we ∈F with¯being the average of the STFT coefficients over the microphones, { } ∈M . Both the magnitude and IPD features are normalized at the window level to reduce random fluctuations, whose effectiveness was experimentally shown in [8].

Speech separation model
Now, we turn our attention to the modeling side of this work. The objective is to construct a speech separation model that receives the feature sequence ( ) ∈T from all the input channels and generates TF mask over the input segment, where denotes the source index and takes a value in {0, · · · , 3}. The first two sources (i.e., ∈ {0, 1}) correspond to the two speakers. The other two sources are used to capture the stationary ( = 2) and transient ( = 3) noise. This is inspired by the SSN architecture proposed in [5], where SSN stands for "speech, speech, and noise". It is required that the output be invariant to the input channel permutation and that the model be applicable to any value.
Several previous studies applied a split-apply-combine (SAC) approach. That is, the same speech separation model is applied to each input channel independently. Then, the TF masks obtained from all the channels are combined, for example, by taking their average [10,11]. The output mask order may be realigned across the channels before the average computation [24]. This approach has two drawbacks. First, it does not take advantage of the nonlinear correlation that may exist between the features of different channels. Second, applying the same model to each channel intensifies the computational cost as the number of microphones increases.
VarArray was designed to overcome the shortcomings of the SAC approach. Figure 1 shows the block diagram of the model. As with SAC, a conformer block is applied to each channel independently at each layer. However, the VarArray model has two key differences to address the two problems mentioned above.
First, TAC layers are interspersed between the conformer blocks. TAC aggregates and fuses the features of all input channels in a non-linearly transformed space and feed the outcome back to the individual channels to combine the information across the channels efficiently in a permutation-invariant fashion. We use a slightly different version of TAC than the original proposal [14]. Specifically, denoting the input sequence of the th channel by ( in ) ∈T , we compute the output for this channel, ( out ) ∈T , as follows: where matrices and are linear transforms. Secondly, instead of maintaining the channels all the way to the last layer, we merge them at an earlier layer. In the example shown in Fig. 1, the model has only one stream after the third conformer block. This significantly reduces the network computation cost.

End-to-end optimization
The VarArray model can be further optimized with respect to an ASR-based loss function. This is possible with the multi-input multioutput architecture of [25], which proposed to combine the speech separation TF mask estimator, MVDR beamformer, feature transformation, and sequence-to-sequence (S2S) ASR model to create an E2E neural network model and perform back-propagation all the way down to the speech separation model. We start the E2E optimization with a VarArray model pretrained with simulated multi-channel data. During training, we minimize the uPIT-style cross-entropy loss between two predicted hypotheses and reference transcriptions, with the back-end ASR model parameters frozen. This allows real transcribed meetings to be used to fine-tune the model parameters and mitigate the mismatch between the simulation and test conditions.
To make the E2E model equivalent to the CSS scheme, we add the sliding window stitching process (see Section 2.1), TF mask sparsification, and gain adjustment (see Section 2.2 for both) to the model. The forward pass processes each multi-channel training sample as follows. The training sample is first fed to the VarArray speech separation model to generate the four TF masks described in Section 3.2, and they are sparsified by retaining only the most dominant mask. Then, two speech separated signals are computed by using the original multi-channel signal and the MVDR filters derived from the sparse TF masks. The generated signals are then chunked into overlapping windows, and the gain adjustment is applied to each windowed signal to equalize the beamformed signal magnitudes with those of the signals that would be obtained by directly applying the TF masks to the input signal. The windowed beamformer signals are stitched together to form the two-channel separated signals that are fed to an ASR model. We used the attention-based encoder-decoder model of [26] in our experiments. Note that these additional elements, especially the gain adjustment, are critical as our training set contains many single-talker samples.

EXPERIMENTAL RESULTS
We conducted experiments to measure the impact of the proposed array-geometry-agnostic speech separation model on the word error rate (WER) of a meeting transcription system. Unlike conventional practices of using signal-based metrics or the WER for pre-segmented speech mixtures, this allowed us to capture the practical usefulness of the speech separation model. Our meeting transcription system, the evaluation tasks, and the results are described below.

System
We employed the audio-visual meeting transcription system of [6]. The vision and speaker diarization modules of the original system were removed to focus on the ASR accuracy. The processing flow was as follows. First, multi-input multi-output dereverberation was performed in real time with a weighted linear prediction method [27]. Then, CSS was applied by using the proposed speech separation model to generate two output signals that were deemed overlap-free. A 1.6-second window was used with a shift of 0.4 seconds, i.e., the stitching was performed by using each 1.2-second overlapping segment. Finally, ASR was applied to these signals independently, and the output transcriptions were merged. Microsoft's internal system was used for ASR.
The speech separation training set consisted of 438 (219 × 2) hours of speech mixtures that were synthesized based on clean speech signals, noise samples, and room impulse responses (RIRs). The simulation was carried out by following the recipe of [6] to create a balanced mix of the single-and two-speaker training samples as well as the four overlap patterns described in [8]. The source speech signals were taken from WSJ SI-284. The RIRs were generated by the image method [28]. Both isotropic and directional noise signals were added to the reverberant speech. The isotropic noise samples were synthesized with the algorithm of [29] while the directional noise data were taken from MUSAN [30]. The RIRs and the isotropic noise samples were generated for two microphone array geometries, namely AMI and MS (see Section 4.2). For each geometry, 219 hours of audio were generated (which is why the size of the first training set was 438 hours).
The VarArray model was configured as follows. As shown in Fig.  1, our model consisted of conformer blocks and TAC layers. Each conformer block comprised five conformer layers, each with four attention heads, 64 dimensions, and 33 convolution kernels. There were a total of five conformer blocks, three of which were applied to each stream in parallel. For TAC, matrices and of Eq. (1) halved the number of dimensions for the TAC layer to preserve the dimensionality. During training, the number of channels (and the microphones to be used) were randomly picked between 3 and 7 for each mini-batch to expose the model to a variety of array geometries and inter-microphone spacing patterns. The training was performed with a mini-batch of 48 four-second sequences.

Tasks
The evaluation was carried out by using two multi-microphone meeting transcription tasks: AMI (or, more precisely, AMI-MDM) and an internal meetings collection, dubbed as MS. For AMI, the "fullcorpus ASR" partition [31] was employed. The audio data were recorded with an eight-channel circular microphone array with a radius of 10 cm. Eight-channel and four-channel conditions were considered, where the latter used only the microphones with odd number indices. MS is an extension of the test set used in [6]. The same seven-channel microphone array was used. Six microphones were arranged in a circle with a radius of 4.25 cm, and the last microphone was located at the center. Seven-channel and three-channel conditions were considered, where the latter used the three microphones including the center microphone and constituting a small equilateral triangle. A total of 60 sessions were recorded (150K words in the reference transcriptions). The WER was estimated with asclite for AMI while our internal scoring tool was used for the MS test set. We used MS to tune system's hyper-parameter values. The AMI development set was actually used as an "evaluation set". Table 1 shows the WERs of the proposed speech separation method and three baseline systems. The first baseline system (BF1) performed super-directive single-output beamforming with real-time beam-steering using our internal beamformer, followed by ASR and thus could not handle speech overlaps. This beamformer performed was as effective as BeamformIt [32] in our internal test. The second baseline system was based on SAC and applied a single-stream separation model to each input stream of Fig. 1 with the magnitude and IPD pair as input to obtain TF masks from each stream. As with [24], the output speech orders were aligned across the streams, and then the TF masks were averaged to be used for beamforming. This separation model had the same number of conformer blocks as our proposed VarArray system. It was trained on the same data set as ours by randomly choosing the input stream. The third baseline system used a concatenation of the magnitude spectra of the first microphone and the IPDs between the first microphone and the rest of the microphones. The model was trained for AMI and MS separately based on the corresponding subsets of the training set due to the input dependency on the number of microphones. For all the four conditions, our system significantly outperformed the second baseline system, showing VarArray's capability of effectively leveraging the spatial information for different microphone array geometries. The proposed model also outperformed the arraygeometry-dependent models for both test sets. This could be largely attributed to the fact that the array-geometry-agnostic model was able to take advantage a larger training set.

Results
Two additional VarArray models were further trained to examine the performance dependency on the array configuration of the training set. One model was trained on the subset that was based on the AMI geometry RIRs. The other model used the MS-geometry portion of the training set. Table 2 shows the experimental results. It was observed that the separation model trained with both microphone array configurations performed the best for both test sets. Nonetheless, the other two models also showed good generalization ability to unseen array geometries. It is noteworthy that the model trained on the AMI-geometry subset performed slightly better than the one trained on the MS-geometry subset. We presume that this is because the AMI-geometry portion of the training set had greater diversity in terms of the inter-microphone distances. Table 3 shows the E2E optimization experiment results for the AMI development and evaluation sets. We segmented the original long-form AMI training recordings by silence positions, removed the segments shorter than 10 seconds, and picked only the segments consisting of one or two speakers, which resulted in a 8.52-hour training set. We experimented two seed models: one trained on the MS-geometry data, one on the AMI-and MS-geometry data. Note that, in this experiment, we used a larger pool of clean speech than in the previous experiments, including samples from our internal data, while we used the same RIR and noise data. The training set size for the seed models was 1,500 hours for the MS-and AMIgeometry portions each. Therefore, the WER numbers are slightly different from those of the previous experiments. Consistent WER gains were observed, suggesting the E2E optimization using the small amount of real transcribed data to help mitigate the mismatch between the training and test conditions. Larger gains were observed for the model pre-trained on the MS-geometry data, indicating that the E2E optimization addressed both the geometry difference and the mismatch between the simulated and real data. It is also noteworthy that the gains were more prominent for non-overlap regions. This could mean that the E2E fine-tuning made the separated signals more friendly to ASR rather than removing interfering signals.

CONCLUSION
We described VarArray, an array-geometry-agnostic speech separation model. Extensive evaluation was conducted with a CSS framework in two real meeting transcription tasks. The proposed model outperformed the SAC approach by efficiently leveraging the spatial information. It was also shown that the VarArray model could generalize to unseen array geometries well and that the E2E optimization could mitigate the mismatch between the training and test conditions. The array-geometry-agnostic modeling is useful for production. In parallel to this work, we examined its impact on personalized noise reduction [33]. Further investigation in different tasks is desired.


text:
A Model You Can Hear: Audio Identification with Playable Prototypes

Machine learning techniques have proved useful for classifying and analyzing audio content. However, recent methods typically rely on abstract and high-dimensional representations that are difficult to interpret. Inspired by transformation-invariant approaches developed for image and 3D data, we propose an audio identification model based on learnable spectral prototypes. Equipped with dedicated transformation networks, these prototypes can be used to cluster and classify input audio samples from large collections of sounds. Our model can be trained with or without supervision and reaches state-of-the-art results for speaker and instrument identification, while remaining easily interpretable. The code is available at: https://github.com/romainloiseau/a-model-you-can-hear

INTRODUCTION
The emergence of deep learning approaches dedicated to audio analysis has led to significant performance improvements [1,2]. Although these methods take sound excerpts as input, they typically rely on high-dimensional latent spaces, making the interpretation of their decisions difficult and limiting the insights gained on the considered data. Furthermore, such methods typically require large corpora of annotated sounds for supervision. We take a different approach, inspired by the recent Deep Transformation-Invariant (DTI) clustering [3] framework which learns prototypes in input space to analyze images or 3D point clouds [4]. Each prototype is associated with a set of transformations (e.g. rotations or morphological transformations) learned in the manner of Spatial Transfor-  mation Networks [5]. The DTI clustering model optimizes a reconstruction loss and associates each input with the prototype that provides the best reconstruction. We propose adapting this approach to the audio domain. The prototypes become log-scaled, Mel-frequency spectrograms, and their associated learnable transformations correspond to plausible sound alterations. We also introduce an additional loss based on cross-entropy in the supervised setting, which we demonstrate can boost the classification accuracy while preserving some interpretability.

RELATED WORK
Audio Classification. Musical instrument and speaker identification are two of the standard tasks used to benchmark audio classification models. Although early musical instrument identification methods focused on distinguish- Figure 2: Method overview. Given an input sound, we predict for each prototype a gain, a pitch shift, as well as low and high frequency filters at each timestamp to generate the output. Prototypes and transformations are learned jointly using a reconstruction loss in either a supervised or unsupervised setting.
ing individual tones achieve appreciable precision, support vector machines operating on spectro-temporal sound representations can reach almost perfect accuracy. Similarly to supervised classification of instruments that play individual notes that could be considered solved [9,10], speaker identification is mostly handled by convolutional and/or recurrent neural networks [11], which achieve almost perfect performances [12]. However, these models rely on complex and abstract latent representations that are not easily interpretable and cannot be visualized as spectrograms or heard in the audio domain.

Prototype-Based Methods.
Auto-Encoders [13,14] learn a compact latent representation of an input sample and are trained using a reconstruction loss. Using regularizations and constraints, the latent space can be adjusted for various tasks, such as classification [15]. However, the learned features remain largely non-interpretable and abstract. Inspired by the literature on images [16], the authors of [17] propose to generate prototypes in the latent space and analyze their similarity to the encoded input samples using a frequency-dependent measure. Although the latent prototypes can be decoded into input space, their role in class prediction remains obfuscated, as all similarities are mixed with a learned linear layer.

Transformation-Invariant Modeling.
Deep transformation-invariant clustering [3] learns explicit prototypes in input space. Each prototype is equipped with dedicated transformation networks, allowing a small set of prototypes to faithfully represent a collection of samples. The resulting models can be used for downstream tasks such as classification [3], few-shot segmentation [4], and even multi-object instance discovery [18]. Jaderberg et al. [5] also proposes learning differentiable transformations in the input space, and feed the transformed data to a classification network. We propose to extend these ideas to the audio domain by learning prototypical log-Mel spectrograms along with adapted transformations.

METHOD
We consider a set of N audio clips x 1 , · · · , x N ∈ R F ×T , each characterized by their log-Mel spectrogram with T time steps and a spectral resolution of F bins. The samples are annotated with class labels y 1 , · · · , y N ∈ Y with Y a given class set {1, · · · , K}. Our goal is to learn an interpretable model that can perform classification and clustering for the audio sample collection x 1 , · · · , x N . Our model is based on prototypical spectrograms equipped with spectral transformation networks, which we present in Sec. 3.1. We propose an unsupervised training scheme in Sec. 3.2, and a supervised setting in Sec. 3.3. Finally, we provide details on parameterization and training of the model in Sec. 3.4.

Sound Reconstruction Model
Our model consists of K prototypes P k ∈ R F directly interpretable in the spectral domain, each equipped with a set of dedicated transformation networks (See Figure 2). Both prototypes and networks are jointly trained to reconstruct input samples. We can then use the reconstruction error as a measure of the compatibility between a sample and a given prototype.

Spectral Transformations Model.
Even a single instrument or speaker cannot be meaningfully characterized by a single sound. In fact, they are typically capable of producing a rich and varied set of sounds. We propose equipping each prototype with a set of transformation networks that learn to change specific characteristics of the prototype, such as its amplitude and pitch, so that it can faithfully approximate a wide variety of samples. This also allows the prototypes to learn more subtle audio attributes, such as timbre or intonations.
Transformation networks associated to each prototype P k take as input an audio sample x ∈ R F ×T as input and predicts a spectral transformation for each time step t in [1, T ]. For each prototype k, we propose a set of transformations specifically designed for the audio domain, illustrated in Figure 3: • Amplification. Given an audio sample x, we predict a gain G k (x) ∈ R T for all time step t. The transformation T gain g (m) adds the gain g to all frequencies of a log-Mel spectrogram m ∈ R F .
• Pitch Shifting. We map an audio sample x to a pitch shift S k (x) ∈ R T for all time step t. The transformation T pitch s (m) shifts the log-Mel spectrogram m ∈ R F by s. We use linear interpolation for handling non-integer shifts.
• High-and Low-Frequency Filter. Given an audio sample x, we define for each time t an affine low-frequency filter L k (x)[t] ∈ R F by predicting its slope and cutoff frequency. Similarly, we predict a high-frequency filter These transformations can be written as follows for a log-Mel spectrogram m ∈ R F and a mel-frequency f ∈ [1, F ]: T high with Φ(f, s) = A log 10 1 + s 10 f /A − 1 the melaware pitch shifting, with A = 2595 [19]. Our transformations have links with moment matching methods [20,21]. In fact, modulating the amplitude, pitch, and spectral support of spectrograms is similar to matching the moment of zeroth order (amplitude gain), first order (pitch shift), and second order (frequency filters).
Reconstruction Model. For each prototype k and each time step t, we successively apply the gain, pitch shift, lowpass, and high-pass transformations in this specific order. The resulting reconstruction R k (x) of an input sound x by a prototype P k is defined as follows: where R k (x)[t] denotes the t-th timestamp of the reconstruction, and • denotes the composition of transformations. Note that the chosen transformations are constrained and limited on purpose, as we want each prototype to represent a restricted and consistent set of audio samples.
We measure the quality of the reconstruction R k (x) of an input sample x by the k-th prototype as the spectrotemporal average of their 2 distance: Note that since this function is differentiable and continuous, it can be used as a loss function to train prototypes and transformation networks as defined in the next sections.

Unsupervised Training
We propose an unsupervised learning scheme in which we do not have access to labels y 1 , · · · , y N during training.
Our goal is to cluster the samples x 1 , · · · , x N into meaningful groups of sounds that can be well approximated by the same prototype spectrogram and its associated transformations. We can train our model by minimizing the following loss: This loss is the sum of the reconstruction errors with optimal cluster assignment and is a classical clustering objective used by K-means-based methods.

Supervised Training
In the supervised setting, each prototype and associated transformation networks are dedicated to a class and trained to reconstruct the samples from this class only.
Once trained, the model predicts for an input x the class of the prototype with the best reconstruction. To promote better class discrimination, we propose a dual reconstruction-prediction objective. We associate each input x with a probabilistic class prediction defined as the softmax of the negative reconstruction error between prototypes. For an input x of label y, we define the prediction loss L ce : where β is a learnable parameter. This loss encourages each prototype to specialize in reconstructing the samples of its associated class. To train our network, we use a weighted sum of both losses: with λ ce an hyperparameter set to 0.01.

Parameterization and Training Details
Architecture. We implement the functions G k , S k , L k and H k as U-Net style neural networks [23] operating on SOL [6,7] LibriSpeech [8]   the log-Mel spectrograms x ∈ R T ×F . To save parameters, all networks share the same encoder, defined as a sequence of 2-dimensional spectro-temporal convolutions and pooling operations. This encoder produces a sequence of feature maps z 0 , · · · , z R with decreasing temporal and spectral resolutions T /2 r × F/2 r . The spectral dimension of these maps is then collapsed using learned pooling operations:z r is of resolution T /2 r . Each decoder is then defined as a sequence of 1-dimensional temporal convolutions and unpooling operations. The spectral maps z 0 , · · · ,z R are used through skip-connections in the decoders. The prototypes P 1 , · · · , P K and the inverse temperature β are directly trainable parameters of the model.

Curriculum Learning.
We ensure the stability of our model by gradually increasing its complexity along the training procedure. First, we learn the raw prototypes without any transformation. Once the training loss does not decrease for 10 straight epochs, we add the gain transformation. We then add the pitch shift and spectral filters successively following the same procedure. Implementation details. We use the ADAM [24] optimizer with a learning rate of 10 −4 , and a weight decay of 10 −6 for the transformation networks and 0 for β and the prototypes. Our model has 545k parameters with K = 128 prototypes. For comparison, the reconstruction model proposed by Zinemanas et al. [17]

EXPERIMENTS
We evaluate our approach in both supervised and unsupervised settings and on two datasets.
Datasets. We consider the following datasets: • SOL [6,7]. This dataset contains 24 450 samples of individual notes played with various playing techniques by 33 different instruments and sampled at 44.1kHz. We evaluate instrument classification.
• Librispeech [8]. This 1000-hour corpus contains English speech sampled at 16kHz. We selected the 128 predominant speakers from the train-clean-360 set. We evaluate the classification of speaker. For both datasets, we randomly selected 70% of the clips for training, 10% for validation, and 20% for testing. We always use as many prototypes as the number of classes in the dataset.

Metrics.
To assess the quality of classification models, we report the following metrics: • Overall Accuracy (OA). Percentage of input samples correctly classified by our model, i.e., best reconstructed by the prototype assigned to their true class.
• Average Accuracy (AA). Average of the classwise accuracy, computed across classes without weights.
• Reconstruction error (L rec ). To assess the quality of the reconstruction, we also report the reconstruction error L rec .
Methods trained in the unsupervised setting cluster the Bassoon Horn Violin Contrabass Accordion Figure 4: Learned Meaningful Prototypes. Comparison between the average spectrogram from one class (left), and the prototypes learned in the supervised setting without (middle) and with (right) cross-entropy loss. Our model can learn spectral representations that are crisper and more discriminative than a simple average. audio samples instead of classifying them. We associate each cluster with the majority class of its assigned training samples. We can then predict for each test sample the class of its best-fitting cluster, allowing us to compute all the classification metrics.
Baselines. To put the performance of our method in context, we trained different baselines: • Classification. We trained a temporal convolutional network to classify log-Mel spectrograms. We supervise this network with the cross-entropy on the labels. This network, which does not provide a reconstruction error, is called Direct Classification. We also evaluated the AP-Net [17] approach on our datasets. However, this method is limited in its interpretability, as it relies on latent prototypes and uses a fully connected layer to classify samples based on their similarity to the prototypes.
• Clustering. We trained a two-dimensional convolutional autoencoder without skip-connections to reconstruct log-Mel spectrograms. We supervised this network with the 2 norm between the input and the reconstruction. We then use K-means [22] on the innermost feature maps to cluster the samples. We also use K-means directly on the predicted similarities between feature maps and APNet prototypes [17]. Note this last approach requires to first train APNet in a supervised setting.

Quantitative Results
Reconstruction. As can be seen in Table 1, each successive addition of a transformation decreases the reconstruction error L rec and improves the accuracy of classification. This shows that the curriculum training scheme allows our model to learn insightful and complementary spectral transformations. Supervised with the cross-entropy loss, our model reaches high accuracy, at the cost of a slightly higher reconstruction error. In fact, the additional objective not only encourages the reconstruction models R k to provide a good reconstruction for the samples of their assigned class, but also a worse reconstruction for other classes. Finally, we see that while adding frequency-filters yields better reconstruction, they hurt classification per-formance on SOL [6,7] in the unsupervised setting. We hypothesize that the filters makes the transformation too expressive, reducing the specificities of the prototypes.
Clustering. As shown in Table 2, our method performs well compared to the baselines. For SOL, our method is affected by the class imbalance between samples, but still produces competitive results. In particular, note that our approach does not need any labels during training, in contrast to the APNet-based approach [17]. For Lib-riSpeech, our method produces convincing clustering results for speaker identification.
Classification. As shown in Table 3, our method reaches almost perfect precision for both datasets. While AP-Net [17] reaches better reconstruction scores, this method learns powerful transformations in an abstract latent space. This leads to a model that is more expressive but less interpretable. In contrast, we restrict the scope of the spectral transformations so that a prototype relates in a meaningful way to the samples that it reconstructs well. Yet, our model is still capable of producing faithful reconstructions, which would be suitable for audio generation tasks.

Interpretability
As shown in Figure 5, our model learns to reconstruct complex input samples (e.g. different notes, techniques, or voices) by automatically adjusting the amplitude, pitch, and spectral support of spectral prototypes. As seen on Figure 4 and Figure 1, these prototypes capture rich spectral characteristics such as timbre, and can be interpreted as an "acoustical fingerprint". Furthermore, as the prototypes are given in the spectral domain, they can be easily played and analyzed. Audio examples featuring the learned prototypes are provided in the supplementary materials.
In Figure 6, we represent the reconstruction of sounds played by various instrument with different prototypes. While the reconstructions are clearly more faithful when using the instruments' dedicated prototype, this opens the perspective for further MIR application such as interpretable timbre transfer.

CONCLUSION
We presented a novel approach to audio understanding by representing large collections of audio clips with few spectral prototypes equipped with learned transformation networks. Our model produces concise, expressive, and interpretable representations of raw audio excerpts that can be used to obtain state-of-the-art results for audio classification and clustering tasks.
Acknowledgements. This work was supported in part by ANR project READY3D ANR-19-CE23-0007 and was granted access to the HPC resources of IDRIS under the allocation 2022-AD011012096R1 made by GENCI. We thank Theo Deprelle, Nicolas Gonthier, Tom Monnier and Yannis Siglidis for inspiring discussions and feedbacks.