summaries: The paper studys cost-sensitive hierarchical classification problems. The novelty is very limited, the experiments are vey tirial. I recommend to reject.

summaries: The authors propose a new framework for cost-sensitive hierarchical classification. First, they decompose it into level by level learning to abstain (with different abstain costs per class) sub-problems. To solve these subproblems, authors apply deep distributionally robust learning (DRL) approach that directly minimizes the abstaining loss (based on Fathony et al. 2018). These two elements create a method named the Layer-wise Abstaining Loss Minimization method (LAM). The proposed method is compared with DARTS on two datasets and achieves attractive performance. The authors also demonstrate that this decomposition makes it easier to achieve the desired performance profile by adjusting abstaining losses of the layers.

summaries: The paper presents an approach for hierarchical cost sensitive classification in which abstentions are allowed. It shows a bijection between original cost sensitive problem and the set of layer wise abstaining losses. It is based on using the existing distributionally robust cost sensitive classification and extending to it to the hierarchical setup. The proposed methodology is demonstrated on birds and cell classification datasets, which are claimed to be large-scale. It is compared to a relatively old DARTS method from 2012.

summaries: The submission study the problem of cost-sensitive hierarchical classification (CSHC) with given label taxonomy via learning to abstentions each layer within the hierarchy. Indeed,  1. CSHC subject has had many researches while using abstention or reject decision has also had many works in flat classification, briding both seems relatively few for which the authors present LAM to achieve a so-called new method.  2. Using DRL framework to solve the learning to abstain problems in each layer makes optimization almost decouplable layer by layer, the rationale behind its strategy should attribute to the proved bijective correspondence between the hierarchical cost-sensitive loss and the set of abstaining losses. 3. LAM achieves better performance on limited benchmarks.

abstractText: We study the problem of cost-sensitive hierarchical classification where a label taxonomy has a cost-sensitive loss associated with it, which represents the cost of (wrong) predictions at different levels of the hierarchy. Directly optimizing the cost-sensitive hierarchical loss is hard, due to its non-convexity, especially when the size of the taxonomy is large. In this paper, we propose a Layer-wise Abstaining Loss Minimization method (LAM), a tractable method that breaks the hierarchical learning problem into layer-by-layer learning-to-abstain sub-problems. We prove that there is a bijective mapping between the original hierarchical cost-sensitive loss and the set of layer-wise abstaining losses under symmetry assumptions. We employ the distributionally robust learning framework to solve the learningto-abstain problems in each layer. We conduct experiments on large-scale bird dataset as well as on cell classification problems. Our results demonstrate that LAM achieves a lower hierarchical cost-sensitive loss in high accuracy regions, compared to previous methods and their modified versions for a fair comparison, even though they are not directly optimizing this loss. For each layer, we also achieve higher accuracy when the overall accuracy is kept fixed across different methods. Furthermore, we also show the flexibility of LAM by proposing a perclass loss-adjustment heuristic to achieve a performance profile. This can be used for cost design to translate user requirements into optimizable cost functions.

authors:
Kaiser Asif
Wei Xing
Sima Behpour
Brian D Ziebart

title: Adversarial cost-sensitive classification

venue: In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence,

authors:
Rohit Babbar
Ioannis Partalas
Eric Gaussier
Massih-Reza Amini

title: On flat versus hierarchical classification in large-scale taxonomies

venue: In 27th Annual Conference on Neural Information Processing Systems (NIPS

authors:
Rohit Babbar
Ioannis Partalas
Eric Gaussier
Massih-Reza Amini
Cécile Amblard

title: Learning taxonomy adaptation in large-scale classification

venue: The Journal of Machine Learning Research,

authors:
Peng Cao
Dazhe Zhao
Osmar Zaiane

title: An optimized cost-sensitive svm for imbalanced data learning

venue: In Pacific-Asia conference on knowledge discovery and data mining,

authors:
Nontawat Charoenphakdee
Zhenghang Cui
Yivan Zhang
Masashi Sugiyama

title: Classification with rejection based on cost-sensitive classification

venue: In International Conference on Machine Learning,

authors:
Anveshi Charuvaka
Huzefa Rangwala

title: Hiercost: Improving large scale hierarchical classification with cost sensitive learning

venue: In Joint European Conference on Machine Learning and Knowledge Discovery in Databases,

authors:
Jianfu Chen
David Warren

title: Cost-sensitive learning for large-scale hierarchical classification

venue: In Proceedings of the 22nd ACM international conference on Information & Knowledge Management,

authors:
Yu-An Chung
Hsuan-Tien Lin
Shao-Wen Yang

title: Cost-aware pre-training for multiclass costsensitive deep learning

venue: In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence,

authors:
Jia Deng
Jonathan Krause
Alexander C. Berg
Li Fei-Fei

title: Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition

venue: IEEE Conference on Computer Vision and Pattern Recognition,

title: Metacost: A general method for making classifiers cost-sensitive

venue: In Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Jianping Fan
Ji Zhang
Kuizhi Mei
Jinye Peng
Ling Gao

title: Cost-sensitive learning of hierarchical tree classifiers for large-scale image classification and novel category detection

venue: Pattern Recognition,

authors:
Rizal Fathony
Zico Kolter

title: Ap-perf: Incorporating generic performance metrics in differentiable learning

venue: In International Conference on Artificial Intelligence and Statistics,

authors:
Rizal Fathony
Anqi Liu
Kaiser Asif
Brian Ziebart

title: Adversarial multiclass classification: A risk minimization perspective

venue: In Advances in Neural Information Processing Systems,

authors:
Rizal Fathony
Mohammad Ali Bashiri
Brian D Ziebart

title: Adversarial surrogate losses for ordinal regression

venue: In NIPS,

authors:
Rizal Fathony
Kaiser Asif
Anqi Liu
Mohammad Ali Bashiri
Wei Xing
Sima Behpour
Xinhua Zhang
Brian D Ziebart

title: Consistent robust adversarial prediction for general multiclass classification

venue: arXiv preprint arXiv:1812.07526,

authors:
Yarin Gal
Zoubin Ghahramani

title: Dropout as a bayesian approximation: Representing model uncertainty in deep learning

venue: In international conference on machine learning,

authors:
Aditya Gangrade
Anil Kag
Venkatesh Saligrama

title: Selective classification via one-sided prediction

authors:
Yonatan Geifman
Ran El-Yaniv

title: Selective classification for deep neural networks

venue: arXiv preprint arXiv:1705.08500,

title: Selectivenet: A deep neural network with an integrated reject option

authors:
Siddharth Gopal
Yiming Yang

title: Recursive regularization for large-scale classification with hierarchical and graphical dependencies

venue: In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Noah F Greenwald
Geneva Miller
Erick Moen
Alex Kong
Adam Kagel
Christine Camacho Fullaway
Brianna J McIntosh
Ke Leow
Morgan Sarah Schwartz
Thomas Dougherty

title: Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning

venue: bioRxiv,

authors:
Peter D Grünwald
A Philip Dawid

title: Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory

venue: Annals of statistics,

authors:
Chuan Guo
Geoff Pleiss
Yu Sun
Kilian Q Weinberger

title: On calibration of modern neural networks

venue: In ICML,

authors:
Shunxin Guo
Hong Zhao

title: Hierarchical classification with multi-path selection based on granular computing

venue: Artificial Intelligence Review,

authors:
Yangqing Jia
Joshua T Abbott
Joseph L Austerweil
Tom Griffiths
Trevor Darrell

title: Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies

authors:
Leeat Keren
Marc Bosse
Diana Marquez
Roshan Angoshtari
Samir Jain
Sushama Varma
SooRyum Yang
Allison Kurian
David Van Valen
Robert West

title: A structured tumor-immune microenvironment in triple negative breast cancer revealed by multiplexed ion beam imaging

authors:
Salman H Khan
Munawar Hayat
Mohammed Bennamoun
Ferdous A Sohel
Roberto Togneri

title: Cost-sensitive learning of deep feature representations from imbalanced data

venue: IEEE transactions on neural networks and learning systems,

authors:
Matjaz Kukar
Igor Kononenko

title: Cost-sensitive learning with neural networks

venue: In ECAI,

authors:
John Langford
Alina Beygelzimer

title: Sensitive error correcting output codes

venue: In International Conference on Computational Learning Theory,

authors:
Kibok Lee
Kimin Lee
Kyle Min
Yuting Zhang
Jinwoo Shin
Honglak Lee

title: Hierarchical novelty detection for visual object recognition

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Haifeng Li
Xin Dou
Chao Tao
Zhixiang Hou
Jie Chen
Jian Peng
Min Deng
Ling Zhao

title: Rsi-cb: A large scale remote sensing image classification benchmark via crowdsource data

venue: arXiv preprint arXiv:1705.10450,

authors:
Ziyin Liu
Zhikang Wang
Paul Pu Liang
Russ R Salakhutdinov
Louis-Philippe Morency
Masahito Ueda

title: Deep gamblers: Learning to abstain with portfolio theory

venue: Advances in Neural Information Processing Systems,

authors:
Daniel A Spielman
Shang-Hua Teng

title: Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time

venue: Journal of the ACM (JACM),

authors:
Han-Hsing Tu
Hsuan-Tien Lin

title: One-sided support vector regression for multiclass cost-sensitive classification

authors:
Grant Van Horn
Oisin Mac Aodha
Yang Song
Yin Cui
Chen Sun
Alex Shepard
Hartwig Adam
Pietro Perona
Serge Belongie

title: The inaturalist species classification and detection dataset

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Haoxuan Wang
Anqi Liu
Junchi Yan
Zhiding Yu
Yisong Yue
Anima Anandkumar

title: Deep distributionally robust learning for calibrated uncertainties under domain shift

venue: arXiv preprint arXiv:2010.05784,

authors:
Hong Wang
Wei Xing
Kaiser Asif
Brian Ziebart

title: Adversarial prediction games for multivariate losses

authors:
Yu Wang
Qinghua Hu
Yucan Zhou
Hong Zhao
Yuhua Qian
Jiye Liang

title: Local bayes risk minimization based stopping strategy for hierarchical classification

venue: IEEE International Conference on Data Mining (ICDM),

authors:
Linjie Yang
Ping Luo
Chen Change Loy
Xiaoou Tang

title: A large-scale car dataset for fine-grained categorization and verification

authors:
Bianca Zadrozny
John Langford
Naoki Abe

title: Cost-sensitive learning by cost-proportionate example weighting

venue: In Third IEEE international conference on data mining,

authors:
Weijie Zheng
Hong Zhao

title: Cost-sensitive hierarchical classification for imbalance classes

venue: Applied Intelligence,

title: Cost-sensitive hierarchical classification via multi-scale information entropy for data with an imbalanced distribution

authors:
Zhi-Hua Zhou
Xu-Ying Liu

title: Training cost-sensitive neural networks with methods addressing the class imbalance problem

venue: IEEE Transactions on knowledge and data engineering,

heading: 1 INTRODUCTION

text:
In many real-world machine learning tasks, the class space has a tree-structured taxonomy, e.g. the organization of species. Biologists have built datasets that include labels ranging from coarse-level domains to the most fine-grained species (Van Horn et al., 2018) . Similar hierarchical structures also exist in the classification of cells (Keren et al., 2018), objects (Yang et al., 2015) and remote sensing images (Li et al., 2017).
When there is a label taxonomy, the ground-truth labels correspond to the leaves of the tree and predicting these labels is known as fine-grained classification. In general, we can predict at any level of the tree, and it is natural to incorporate a cost of such predictions. For example, given an image of “owl”, classifying it as either “bird” or “animal” are both correct, but the cost associated with them should be different, since the predicted labels have different levels of specificity. In addition, hard examples are generally easier to distinguish at a coarser level, e.g. in animal mimicry in Fig. 1 (a). In many practical settings, the user is able to specify the cost of different types of errors in a classification problem. The cost can be formulated as an asymmetric matrix with the columns as the labels at the leaves and the rows as the predictions at any level.If we have a label taxonomy, as in Fig. 1 (a), an example of the cost matrix is in Fig. 1 (b).
Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex. Previous works usually rely on its approximation using convex surrogates losses (Zhou & Liu, 2005; Tu & Lin, 2010; Chung et al., 2016; Liu et al., 2019; Khan et al., 2017). However, in hierarchical classification, a convex surrogate for the entire taxonomy is not easy to find. Previous work on hierarchical classification usually takes a top-down strategy to decouple the problem into O(n) binary classification problems for a n node tree (Fan et al., 2015; Charuvaka & Rangwala, 2015; Chen & Warren, 2013). However, there is no guaranteed way to optimize the resulting cost-sensitive loss. The most relevant work that deals with both cost-sensitive and hierarchical learning takes a reward maximization perspective and
Details can be found in Sec. 3. (b) The corresponding cost-sensitive loss Ch on the full hierarchy.
uses a post-training heuristic to find the desired trade-off between accuracy and specificity, given an overall accuracy (Deng et al., 2012). However, it requires an accurate estimation of probabilities (confidence levels) for the leaf nodes, which is not possible in standard deep neural networks. This is because DNNs tend to be overconfident even when they are wrong (Guo et al., 2017).
Distributionally robust learning framework can be incorporated to optimize a given cost-sensitive loss directly (Asif et al., 2015), which involves solving a linear programming problem at each inference step. However, this scales poorly as the number of classes grows since the time of solving linear programs numerically scales polynomially with the number of classes (Spielman & Teng, 2004). Instead of using a numerical solution, analytical solution to the linear program is available for certain special classes including the abstaining losses when the loss value is no greater than 0.5 (assuming loss is in [0, 1]) (Fathony et al., 2016; 2018). Abstaining means that the predictor rejects to predict any of the classes and instead pays a given abstaining cost, which is typically not higher than the cost of wrong prediction. However, these special cases of cost sensitive losses do not incorporate a hierarchy. Moreover, most of the previous methods only discuss the case of low-dimensional linear features and do not incorporate deep neural networks (Asif et al., 2015; Fathony et al., 2016; 2018).
Our approach: In this paper, we take a bottom-up approach to break down the hierarchical costsensitive loss into the abstaining losses at each layer. Here, abstaining means that the predictor decides not to predict the leaf nodes and instead explores predicting labels at higher levels of the tree. The key insight is that each mistake made by the predictor is due to a series of abstaining and prediction decisions along the hierarchy, which results in a cumulative cost. Fig. 1 (a) demonstrates learning-to-abstain losses in each layer that are equivalent to the hierarchical cost-sensitive loss in Fig. 1 (b). The corresponding prediction mechanism is shown in Fig. 2 (b), where a prediction is first made on the leaves and the abstained data is evaluated on its parents in the tree. We then solve the learning-to-abstain problems in each layer using a distributionally robust learning (DRL) method. Applying DRL to each layer of the tree involves retraining of the neural network for different number of classes. In practice, retraining is efficient since only the heads are replaced and the rest of the representation is shared. Therefore, we train the higher-level models with a different head and parameters from the lower-level ones as the initialization (Fig. 2 (a)).
Our contribution can be summarized as follows: 1. We propose to decompose the cost-sensitive hierarchical classification problem into learning-to-
abstain sub-problems at each layer. We show the bijective correspondence when the cost-sensitive losses satisfy symmetry conditions (Sec. 3.1).
2. We propose to solve the learning-to-abstain problems using a distributionally robust learning (DRL) approach that directly minimizes the abstaining loss at each of the layers. Our method also allows for efficient training of the model in different layers. We call our method Layer-wise Abstaining Loss Minimization method (LAM).
3. We conduct experiments on large-scale bird recognition with 244 nodes (iNat2017 (Van Horn et al., 2018)) and cell classification tasks (Keren et al., 2018). Our results demonstrate that LAM achieves lower hierarchical cost-sensitive loss in high accuracy regions, compared to previous
methods, even though it is not directly optimizing this loss. This is because the regime of high accuracy corresponds to low abstaining costs and our method is proven to correspond to the original hierarchical loss in this regime. For each layer, we can also achieve higher accuracy when the overall accuracy is kept fixed across different methods.
4. Our method is also flexible enough to achieve a user-defined performance profile efficiently. We provide a heuristic based on per-class loss adjustment to show the potential of our method in achieving arbitrary performance profiles.

heading: 2 BACKGROUND

text:
Hierarchical Classification Given input x 2 X and labels y 2 Y , we are interested in learning a classifier f(x) : x ! y. We know that y has a taxonomy as a tree structure, which means each x can be classified as a set of correct labels, from fine-grained to coarse labels. Assume the most fine-grained class among all the correct labels for x is y⇤(x), then as long as f(x) 2 P (y⇤(x)), the prediction is considered as correct, where P (y⇤(x)) represents the “path” from the leaf node y⇤(x) to the root of the tree. The classifier predicts a coarse label by abstaining from the more fine-grained leaf nodes. However, not all correct labels provide the same correctness. The user defines a cost matrix C that represents the importance of the different types of errors and abstentions when evaluating the performance of f(x). Assume we have m most fine-grained level classes and n total labels in the tree. The matrix has m columns and n rows. The (i, j)-th entry C(i, j) in the cost matrix represents the cost the classifier suffers when the ground-truth label of x is j but the prediction is i.
Cost-sensitive Learning Assume our training data is drawn from a data distribution P (x, y), cost-sensitive learning aims to find a classifier that minimizes the following expected loss function: f = arg minE(x,y)⇠P (x,y)[C(y, f(x))]. Under the framework of empirical risk minimization, the expectation is approximated using finite samples. Assume, we have M data points (xi, yi) drawn from training set Dtrain, then the problem is converted to: f = arg min 1M P (xi,yi)⇠Dtrain C(yi, f(xi)). Conventional machine learning usually approximates the expected cost-sensitive loss using a convex surrogate. Different losses induce different classifiers. For example, the 0-1 loss is a special case of the cost-sensitive loss when there is no label taxonomy. Both log loss and hinge loss are convex surrogates of 0-1 loss. Logistic regression and support vector machines are derived by optimizing log loss and hinge loss, respectively.
Learning to abstain Learning-to-abstain problems aim to minimize the abstaining losses using training data. Abstaining loss is a special case of general cost-sensitive losses. It is an asymmetric matrix with k + 1 rows and k columns if k is the total number of classes. The additional row represents the abstaining cost for each of the ground truth labels. We use Ca to denote an abstaining loss. The abstaining cost should be smaller than the cost of making a wrong prediction. Otherwise, the classifier will never abstain. Then if the first k ⇥ k block of Ca is exactly the 0-1 loss, than the entries ↵k needs to satisfy: 0 < ↵k < 1.

heading: 2.1 DISTRIBUTIONALLY ROBUST COST-SENSITIVE CLASSIFICATION

text:
We now introduce the background in distributionally robust learning (DRL) for optimizing the cost-sensitive loss.
The formulation: Fathony et al. (2018) proposed a distributionally robust learning method that incorporate a minimax game to robustly optimize the expected cost-sensitive loss. Assume g(x) and h(x) are valid conditional label distributions. Formally,
G : Rd 7! RC , {g(x)|x 2 X , g(x) 2 RC \ } (1) Here, d, C and denote the input dimension, class number, and probabilistic simplex, respectively.
Instead of empirical risk minimization, the DRL approach optimizes the following: arg min
G
max H2⌃
Ex⇠P (x)[gT (x)Ch(x)], (2)
where g(x),h(x) 2 RC are the conditional label distributions given an input x and G, H are the entire distribution over all the input. ⌃ is a constraint set h needs to also reside in. The idea of this formulation is that g, as a predictor player, plays a two-player adversarial risk minimization game Grünwald et al. (2004) with an adversarial player h. The predictor player minimizes the expected cost, while the adversary player maximizes the expected cost. The adversary is allowed to perturb the ground-truth labels, subject to certain feature-matching constraints to ensure data compatibility. In particular, the adversary H is constrained by the empirical training data features:
⌃ = {H| P ihy (xi) = P iI[yi = y] (xi), 8y}, (3)
where xi ⇠ Dtrain, hy is the y-th dimension of h and is a feature function. Eq. 3 is a necessary but not sufficient condition for H equals to the ground truth P (Y |X), when P (Y |X) is represented using finite training data. The insights here is that given a predefined feature function , when the adversary perturbs the labels, certain aggregate function of on g should equal to the counterpart on the empirical source data.
The derived prediction form: Solving Eq. 2 involves solving a two-player zero-sum game with constraints. Fathony et al. (2018) provides analytical solutions for G when the cost-sensitive loss C takes some special cases. In these cases, G would be parameterized with dual parameters ✓. Here, we omit the derivation but show an example of the analytical solution when we incorporate Ca in Eq. 2 with ↵k  0.5. Theorem 1. (Theorem 13 in (Fathony et al., 2018)) Let ✓⇤ be the learned parameter, and f(x) be the potential vector for all classes where fi(x) = ✓⇤| i(x), i 2 [0, C]. Given a new data point x, let i⇤ = arg maxi fi(x) (break tie arbitrarily), j⇤ = arg maxj 6=i⇤ fj(x), and ei⇤ 2 RC be the i⇤-th cannonical vector. Then the predictor G can be directly computed as, for each x:
g(x) =
 ei⇤
0
if fi⇤(x) fj⇤(x) 1 (4)
g(x) =  (fi⇤(x) fj⇤(x))ei⇤ 1 fi⇤(x) + fj⇤(x) if fi⇤(x) fj⇤(x) < 1. (5)
The parameter learning: Learning the parameters in DRL involves computing the (sub)-gradients of ✓ with respect to the dual objective of Eq. 2. In this paper, we investigate using deep neural networks as the feature function (x) in DRL.

heading: 3 COST-SENSITIVE HIERARCHICAL CLASSIFICATION

text:
In this section, we describe our proposed distributionally robust learning method for cost-sensitive learning in hierarchical classification. We first break the cost-sensitive loss on the full taxonomy into abstaining losses in the learning-to-abstain sub-problems in each layer (Sec. 3.1). We then propose to use deep DRL to solve the learning-to-abstain problems. We provide details on the gradient (Sec. 3.2). Finally, we propose the algorithms for training and evaluating our cost-sensitive hierarchical classification method (Sec. 3.3).
3.1 COST-SENSITIVE LOSS AS AN ACCUMULATIVE LOSS
We take a bottom-up approach and observe that the prediction on the taxonomy takes a series of decisions. In Fig. 1, if an “owl” is classified as a “insect”, it means the classifier abstains on the leaf layer and then makes a wrong prediction in the “bird” vs. “insect” layer. Therefore, we can break the overall classification problem into smaller layer-by-layer learning-to-abstain problems. Here we analyze whether there is a one-on-one correspondence between a Ch and a set of Ca in hierarchical classification.
Given a tree taxonomy with m leaves and n total number of nodes in the label space, we assume P (y) represents the “path” from the leaf node y to the root of the tree. We also assume D(y) represents the index of the layers of each node. Assume the leaf nodes has D = 0. Then we have the following: Lemma 1. Assume an abstaining loss Ca is associated with each of the layers with ↵i as the abstaining loss for each of the nodes, i 2 [1, n 1] (except the root node), a hierarchical costsensitive loss Ch is uniquely constructed by Ch(i, j) = P k2P (j)\k>i ↵k + I(i 2 P (j)).
Lemma 2. Given a tree taxonomy, assume the overall cost-sensitive loss is Ch. Let L denote the set of leaf nodes in the tree. D(y) is the depth of a node. There is a unique Ca for each layer such that Ch(i, j) = P k2P (j)\k>i ↵k + I(i 2 P (j)), if Ch satisfies the following
1. 8y 2 L, if i 2 P (y), Ch(i, y) < mink 62P (y) Ch(k, y);
2. 8i 62 L, Ch(i, j)  1 + D(i);
3. 8i 62 L, Ch(i, j) = Ch(i, j0), for 8j, j0 satisfying i 2 P (j) \ P (j0);
4. 8i 62 L, Ch(i, j) = Ch(i, j0), for 8j, j0 satisfying i 62 P (j) [ P (j0).
Remarks: Lemma 1 is trivial by construction. Lemma 2 can be proved easily by induction. Constraints in lemma 2 defines the structure the cost-sensitive loss needs to possess. Without loss of generality, this lemma shows that our method is applicable to a large set of cost-sensitive losses. Here we explain each of the conditions as follows:
1. The cost of a mistake in any layer is higher than a correct prediction. This is naturally required in any meaningful cost-sensitive losses.
2. The abstaining cost is not greater than the cost of making a mistake in any layer. This makes sure that there is no harm to abstain, compared to predicting a wrong label. Otherwise, no abstention will be made and all predictions and mistakes will be made in the leaf nodes.
3. If two leaf nodes share the same relative position in the tree, i.e., they are the children of the same father, there is symmetry in their cost if they are abstained on. For example, in Fig. 3, if a predictor makes abstention decisions on the leaf layer but predicts node 5, it suffers the same cost when the ground truth is node 2 or 3: Ch(5, 3) = Ch(5, 2).
4. The symmetry also holds if a mistake is made in the non-leaf layers. Following the same logic, in Fig. 3, if a predictor makes abstention decisions on the leaf layer but predicts node 4, it suffers the same cost when the ground truth is node 2 or 3: Ch(4, 3) = Ch(4, 2) .

heading: 3.2 DISTRIBUTIONALLY ROBUST LEARNING FOR LEARNING-TO-ABSTAIN PROBLEMS

text:
We present the distributionally robust learning method for the layer-by-layer learning-to-abstain problems. We first review the (sub-)gradient of the ✓ in the original DRL formulation. We then discuss how to incorporate deep neural networks and how parameters in the neural networks are optimized. Finally, we present an algorithm for optimizing the abstaining loss in each layer in Alg. 1. Parameter learning in DRL: According to Fathony et al. (2018), the subdifferential of the objective with respect to the parameter ✓ can be fully characterized by the following. For each data sample:
s✓j (x, y) = conv n h ⇤ j (x) I(y = j) (x) h(x) 2H⇤ o , (6)
where H⇤ is the set of optimal solution of the following:
h ⇤ = arg max
h2 min g2
n g | Cah + ✓ | hP j hj (x) P j I(y = j) (x) io . (7)
In general, we need solve a linear program to compute h⇤. When ↵  0.5 in Ca, there exists an analytical form of the solution such that the gradient can be computed efficiently.
Incorporation of deep neural networks: Instead of low-dimensional features, we incorporate deep neural networks to learn representation (x,✓,w) for constraining the adversarial h in the DRL framework. The last layer of the network is a linear layer parameterized by ✓, the other layers are parameterized with w. Formally, (x,✓,w) = ✓T (x, w). Then after we update ✓ using Eq. 7 and Eq. 6, the sub-gradient of (x,w) can be computed as follows:
s (x, y) = conv n ✓ · ⇣ h ⇤(x) 1(y) ⌘ h(x) 2H⇤ o , (8)
where 1(y) is the one-hot encoding of y. Then we use Eq. 8 to back-prop to compute gradients of w in the networks. We present the algorthim in Alg. 1.
Algorithm 1 Training for deep DRL for optimizing the layer-by-layer abstaining loss
1: Input: Training data, abstaining loss for this layer Ca, DNN ✓T (x, w), SGD optimizer learning rate , epoch number T . 2: Initialization: epoch 0 3: While epoch < T 4: For each data sample (x, y) 5: Compute h⇤ following Eq. 7; 6: Compute the gradient of ✓ fol-
lowing Eq. 6; 7: Back-propagation to compute
gradients for w. 8: ✓ ✓ s✓(x, y); 9: Update w by SGD2( ) using
derived gradients; 10: epoch epoch +1 11: Output: Trained networks ✓T (x, w).
Algorithm 2 Training and testing of LAM 1: Input: Training data Dtrain, test data Dtest,
cost-sensitive loss Ch, label taxonomy with T layers, T DNN: M0, ...,MT 1. 2: Decompose Ch into T layer-by-layer abstaining loss C0a , ...,CT 1a , pretrain M0 using C0a ; 3: For t 2 [0, T 1]: 4: Use pretrained M0 to initialize
M1, ...,MT 1; 5: Mt = Algorithm 1(Cta, Dtrain); 6: 7: //Making predictions: 8: For t 2 [0, T 1]: 9: (Dabt, Ŷt) = Predict (Mt, Dtest,Cta); 10: Dtest = Dabt; 11: Output: Trained models M0, ...,MT 1,
predictions for all the test data.

heading: 3.3 THE OVERALL ALGORITHM

text:
We now present the training and prediction techniques for the whole taxonomy.
Training: We run Alg. 1 for each layer of the taxonomy using the corresponding Ca in that layer. However, since the only difference in the data is the relabeling of the labels using nodes in different layers, we initialize the networks in the father layers using pre-trained networks in the leaf layer to achieve efficient training.
Making predictions: After the models for each of the layers are trained, the prediction follows a bottom-up procedure. We first use the model in the leaf layer to make predictions on all the samples, the abstained samples are collected for predictions using the father layer’s model. Note that the prediction also requires the loss Ca since it involves solving the game, like in Theorem 1. This process goes on until we reach the root of the tree. Eventually, every data sample is assigned to a node in the taxonomy. We summarize the whole algorithm in Alg. 2.

heading: 4 EXPERIMENTS

text:
In this section, we evaluate the effectiveness of our LAM algorithm in two real-world tasks: bird classification and cell classification. We also compare one-layer LAM with previous baselines on real-world image classification datasets: CIFAR10 and SVHN.
For the bird data, we use a subset of the Aves supercategory data from iNaturalist (iNat) 2018 Competition. The Aves dataset contains images of bird species in the Aves kingdom with labels on the hierarchical taxonomy of the animal in the image: from kingdom to order, family, genus, and species (ground-truth). Thus, there are five layers in the Aves data tree with the Aves kingdom as the
root and species as leaf nodes. The iNat data was pre-split into training and validation images, where the training data exhibits a long-tailed distribution across classes, and the validation set exhibits a uniform distribution across classes. We include more dataset details in the appendix.
Our cell data comes from the triple-negative breast cancer (TNBC) dataset introduced in Keren et al. (2018). This data comes from multiplexed imaging, a technique that quantifies the abundance of biomarkers within a sample while still capturing the spatial organization of the tissue. Table 1 gives the cell types in the dataset and their respective counts in the training and validation sets. Note that in this data, not all the leaf nodes are in the same level. We then add dummy intermediate nodes to the taxonomy,
which does not affect the cost structure. More dataset details can be found in the appendix.
CIFAR10 and SVHN are both non-hierarchical or one layer datasets. We use these datasets to validate our advantage on one-layer learning-to-abstain problems over previous baseline Deep Gambler (Liu et al., 2019). To be consistent, we use the same training, validation, and test split as in Liu et al. (2019).

heading: 4.1 COMPETING METHODS

text: We use the same abstaining loss for each layer and implement LAM models with ↵ = [0.1, 0.2, ..., 0.9] in Aves classification. In cell classification, we set ↵ = [0.05, 0.1, 0.15, ..., 0.3]. We compare our method with DARTS (Deng et al., 2012). DARTS uses the information gain as the reward for each node in the tree. After a model is trained on the leaf nodes, a hyperparameter that adjusts the balance between accuracy and specificity is tuned using binary search. Again, it is not clear what cost-sensitive loss is optimized. In order to make the comparison fair, we implement two versions of DARTS. 1) DARTS (reward): We use the original reward but set the overall accuracy to be the same as every DRL model. 2) DARTS (abstain loss): We modify the reward in DARTS using a “negative cost” so that each node in the tree is associated with a reward that is either ↵ for nodes in the leaf layer or summation of previous abstaining losses for non-leaf layers. Then we tune the modified version of DARTS using the same overall accuracy achieved by our method. Also, sometimes DARTS cannot achieve the exact required accuracy. We then compare to the closest similar accuracy we can find using DARTS. Note that this correspondence between abstaining losses and rewards is only possible when we use the same ↵ for all the nodes in the same layer. We also compare our method against Deep Gamblers (DG) (Liu et al., 2019). DG minimizes an abstaining loss function based on the doubling rate of gambling–inspired by portfolio theory–in an empirical risk minimization framework. We include more details in the appendix.

heading: 4.2 RESULTS AND ANALYSIS

text:
How abstaining loss affects the learning-to-abstain performance? We plot the overall error rate and abstaining rate of LAM models using different abstaining losses in the Aves classification task’s leaf layer in Fig. 4. We see that the larger the abstaining cost is, the more aggressive the classifier is, thus the larger the error rate and the lower the abstaining rate. This validates the working mechanism of the abstaining cost. Imagine the abstaining cost is closer to the cost of making a wrong prediction.
The classifier does not gain much by rejecting to make a prediction. On the other hand, if the abstaining cost is small, the classifier would favor abstentions more.
How does the overall algorithm perform? We show the comparison with DARTS of non-abstained accuracy on the Aves classification in Fig. 5 (a) and (b), which are the correct samples over all the predicted samples. We can see that, by achieving the same overall accuracy in the whole taxonomy, our method achieves higher non-abstained accuracy consistently in each of the layers. We also compare the cost-sensitive loss achieved by our method and the modified version of DARTS in Fig. 5 (c). We can see that our cost is much lower in the high accuracy domain. We also show the comparison with DG of cost-sensitive loss on CIFAR10 and SVHN in Fig. 7 (a) and (b). We see that LAM (with only one-layer) achieves better cost-sensitive loss when the same accuracy is obtained. In Fig 7 (c) and (d), we compare cumulative overall accuracy of equivalent LAM and DG models on Aves classification. The overall accuracy LAM achieves is significantly larger than DG at every level at which a model is trained–when we set DG’s loss to be the same as LAM. Note that level 4 denotes the singular root node of the Aves hierarchy, and all inputs abstained on in level 3 are automatically deemed correct in level 4.
Can our method be used to achieve a specific performance profile? Since our method can assign different abstaining losses to each class (different ↵k), we show how we can achieve a similar performance profile by using a loss-adjustment heuristic. Here, the user-defined performance profile means the tuple of overall accuracy, error rate and abstaining rate. In Fig. 5 (d), we use the leaf layer in the Aves taxonomy and start with ↵ = 0.2 LAM. Our goal is to achieve a similar performance profile achieved by DARTS with overall accuracy 0.9 in the leaf layer. After four iterations of loss-adjusting, we achieve a very similar performance profile with DARTS.
The loss-adjustment method is a binary search that takes as inputs the error per class profiles of the LAM model (in the first iteration, ↵ = 0.2 LAM) and target model (DARTS accuracy 0.9). In every iteration, the LAM model is retrained using an alpha vector that is updated based on a comparison of the error per class between the LAM and target model: if, for a class k, the error rate is less than the error rate of the same class achieved by the target model, then ↵k for that class is increased to be (↵k,current +1)/2. An increase in ↵ corresponds to a lower abstaining and higher error rate as seen in Fig. 4. If the error rate is greater than the error rate achieved by the target model, then ↵ is updated as (↵k,current)/2. The updated ↵ values are
compiled into an updated vector to retrain the LAM model on. This iterative process stops via a ”lookback” condition: at iteration i, if the overall accuracy, error, and abstain rates of the previous iteration i 1 are all closer in magnitude to the target rates than the current iteration, we stop and use the LAM model trained at iteration i 1 as the closest model that achieves the target profile.

heading: 5 RELATED WORK

text:
Deep Cost-Sensitive Learning Traditional cost-sensitive learning works usually focus on transforming data samples (Domingos, 1999), changing the prediction procedure (Langford & Beygelzimer,
2005; Zadrozny et al., 2003), and modifying the existing learning models to include cost-sensitive information (Kukar et al., 1998; Tu & Lin, 2010; Zhou & Liu, 2005; Cao et al., 2013; Gangrade et al., 2021). The investigation of the cost-sensitive perspective in deep learning is still limited. Existing works usually optimize a convex surrogate (Chen & Warren, 2013; Chung et al., 2016; Khan et al., 2017; Charoenphakdee et al., 2021). Learning to abstain can be regarded as a special case of cost-sensitive learning. There are several different strategies. Besides the naive dropout-based method (Gal & Ghahramani, 2016), Geifman & El-Yaniv (2017; 2019) proposed to change the architecture to have a network for deciding abstain or not, while Liu et al. (2019) utilized a tunable weight to incorporate the rejection option in the loss function. In our paper, we investigate cost-sensitive loss on a tree taxonomy and decompose the loss into tractable layer-by-layer abstaining losses, which can be optimized using DRL.
Deep Hierarchical Classification For learning with hierarchical classes, most of the work takes the top-down strategy (Babbar et al., 2013; 2016; Zheng & Zhao, 2020; Guo & Zhao, 2021), which suffers from error propagation. Others seek to add an adjustment component (Fan et al., 2015) or hierarchyrelated regularization (Gopal & Yang, 2013; Charuvaka & Rangwala, 2015) to the model to make reasonable hierarchical decisions. Deng et al. (2012) is a post-processing method that uses posterior probabilities obtained on a validation set to tune parameters according to information-gain reward and target accuracy. These methods heavily depend on an accurate probabilistic classifier, which is usually not trained under using hierarchical information. Direct improvements over DARTS have included Bayesian methods learning (Jia et al., 2013) and integrating the uncertainty of predicting the parent nodes (Wang et al., 2017), as well as an approach that uses modified top-down and flatten methods to recognize novel classes (Lee et al., 2018). Recently, Zheng & Zhao (2021) proposes to decompose the hierarchical problem in a similar way. However, it stills uses the post-hoc thresholding method for each of the layers.
Distributionally robust learning Distributionally robust learning methods have been developed to robustly minimize the general loss functions (Fathony et al., 2016; 2017; 2018). In the most general case, the minimax game is solved using linear programming, which is hard to scale to large number of classes. More advanced techniques have been developed to provide analytical solutions (Fathony et al., 2018) and deal with combinatorially large size of the game (Wang et al., 2015) to optimize the general performance metric. Recently, this framework has been further developed to incorporate deep neural works for optimizing the logarithmic loss under covariate shift (Wang et al., 2021) and make general losses differetiable (Fathony & Kolter, 2020). Our work studies the relation between the general cost-sensitive loss and the learning-to-abstain losses and proposes feasible solutions using the DRL framework to deal with real-world high-dimensional data.

heading: 6 CONCLUSION

text: In this paper, we propose to break the overall cost-sensitive learning problem into learning-to-abstain sub-problems in each layer in the tree taxonomy. We provide a distributionally robust learning method for optimizing the abstaining losses in each layer. We show the bijective correspondence between the hierarchical cost-sensitive loss and the set of abstaining losses. We evaluate our method on real-world datasets from applications in bird classification and cell classification. Our method outperforms the competing methods by achieving lower cumulative cost and higher accuracy in each layer. We also provide a heuristic for adjusting abstaining losses to achieve a desired performance profile.


summaries: This paper proposes a method for improving the detection and the localization of several sound sources using a 4-channel signal (The addressed configuration, determined or under-determined case is not detailed in this paper). The proposed approach is based on a so-called "synperiodic" filter bank representation that is used as the input of a deep convolutional neural network. Finally method proposed by the authors is comparatively evaluated on the DCASE2020 dataset and pretends to provide the best results.

summaries: I am sorry, by mistake I posted the incomplete review version!   This manuscript proposes a sound source localization method utilizing "synperiodic filter" bank representation as to the input of a deep convolutional neural network.  Authors claim that the convolution of the proposed filterbanks with the raw waveform helps to achieve multi-scale perception in the time domain which results in performance improvement.  The authors demonstrated that their method could outperform some of the state-of-the-art similar work.

summaries: This paper addresses the problem of sound event detection and localization from multi-channel raw audio waveforms. Essentially, a different audio front end feature extraction scheme - "Synperiodic Filterbank" if proposed which can be parameterized and jointly learnt along with a backbone classifier in an end-to end manner. The feature extraction scheme proposed is grounded in principle of uncertainty governing time-frequency resolution. Through ablation studies the authors show that this feature extraction scheme performs better (albeit marginally) compared to some of the traditionally used feature extraction schemes like MFCC, and LFBEs. Subsequently, the authors show that combining the proposed "Synperiodic Filterbank" with a significantly large model (~3X times the closest baseline model in terms of No. of trainable parameters) based on the Transformer Architecture offers a dramatic improvement in the results compared to the baseline.

summaries: This submission addresses the sound source detection problem. Compared to previous method, the authors propose a novel synperiodic filerbank compared to syndistance bank, which learns a data-dependent time-frequency resolution map. Experiments show that the proposed method achieves state-of-the-art performance on several datasets.

abstractText: Accurately estimating sound sources’ temporal location, spatial location and semantic identity label from multi-channel sound raw waveforms is crucial for an agent to understand the 3D environment acoustically. Multiple sounds form a complex waveform mixture in time, frequency and space, so accurately detecting them requires a representation that can achieve high resolutions across all these dimensions. Existing methods often cannot do it well because they either extract hand-engineered features (i.e. STFT, LogMel) that require a great deal of parameter tuning work, or propose to learn a single-scale filter bank to process sound waveforms that has limited time-frequency resolution capability. In this paper, we tackle this issue by proposing to learn a group of parameterized synperiodic filter banks. Each synperiodic filter’s length and frequency resolution are inversely related, hence is capable of maintaining a better time-frequency resolution trade-off. By alternating the periodicity term, we can easily obtain a group of synperiodic filter banks, where each bank differs in its temporal length. Convolution of the proposed filterbanks with the raw waveform helps to achieve multi-scale perception in the time domain. Moreover, applying synperiodic filter bank to recursively process a downsampled waveform enables to achieve multi-scale perception in the frequency domain. Benefiting from the multi-scale perception in both time and frequency domain advantage, our proposed synperiodic filter bank group learns a dynamic multi-scale time-frequency representation in a data-driven way. Following synperiodic filter bank group front-end, we add a Transformer-like backbone with two parallel soft-stitched branches to learn semantic identity label and spatial location representation semi-independently. Experiments on both direction of arrival estimation task and the physical location estimation task shows our framework outperforms existing methods by a large margin. Replacing existing methods’ front-end with synperiodic filter bank also helps to improve the performance.

authors:
Sharath Adavanne
Archontis Politis
Joonas Nikunen
Tuomas Virtanen

title: Sound event localization and detection of overlapping sources using convolutional recurrent neural networks

venue: In IEEE Journal of Selected Topics in Signal Processing,

authors:
M.S. Brandstein
H.F. Silverman

title: A robust method for speech signal time-delay estimation in reverberant rooms

venue: In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP),

authors:
J. Bruna
S. Mallat

title: Invariant scattering convolution networks

venue: IEEE Transactions on Pattern Analysis Machine Intelligence (TPAMI),

authors:
Yin Cao
Turab Iqbal
Qiuqiang Kong
Fengyan An
Wenwu Wang
Mark D Plumbley

title: An improved event-independent network for polyphonic sound event localization and detection

title: Xception: Deep learning with depthwise separable convolutions

venue: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Junyoung Chung
Caglar Gulcehre
KyungHyun Cho
Yoshua Bengio

title: Empirical evaluation of gated recurrent neural ntworks on sequence modelling

venue: In Adv. Neural Inform. Process. Syst.,

authors:
Steven Davis
Paul Mermelstein

title: Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences

venue: IEEE Transactions on Acoustics, Speech, and Signal Processing,

authors:
Francois Grondin
James Glass
Iwona Sobieraj
Plumbley Mark D

title: A study of the complexity and accuracy of direction of arrival estimation methods based on gcc-phat for a pair of close microphones

venue: In Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Sun Jian

title: Identity mappings in deep residual networks

venue: In European Conference on Computer Vision (ECCV),

authors:
Yuhang He
Long Chen
Jianda Chen

title: Multi-task relative attribute prediction by incorporating local context and global style information

venue: In Proceedings of the British Machine Vision Conference (BMVC),

authors:
Yuhang He
Andrew Markham
Niki Trigoni

title: Sounddet: Polyphonic sound event detection and localization from raw waveform

venue: In International Conference on Machine Learning (ICML),

authors:
Yedid Hoshen
Ron J. Weiss
Kevin W. Wilson

title: Speech acoustic modeling from raw multichannel waveforms

venue: In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Navdeep Jaitly
Geoffrey Hinton

title: Learning a better representation of speech soundwaves using restricted boltzmann machines

venue: In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Alex Kendall
Yarin Gal
Roberto Cipolla

title: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

venue: In Computer Vision and Pattern Recognition

authors:
Taejun Kim
Jongpil Lee
Juhan Nam

title: Comparison and analysis of samplecnn architectures for audio classification

venue: IEEE Journal of Selected Topics in Signal Processing,

authors:
Diederik Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: In 3rd International Conference on Learning Representation (ICLR),

authors:
Tsung-Yi Lin
Michael Maire
Serge Belongie
James Hays
Pietro Perona
Deva Ramanan
Piotr Dollár
C. Lawrence Zitnick

title: Microsoft coco: Common objects in context

venue: European Conference on Computer Vision (ECCV),

authors:
Tsung-Yi Lin
Piotr Dollár
Ross Girshick
Kaiming He
Bharath Hariharan
Serge Belongie

title: Feature pyramid networks for object detection

venue: In IEEE/CVF Conference on Computer Vision and Pattern Recognition

authors:
Wei Liu
Dragomir Anguelov
Dumitru Erhan
Christian Szegedy
Scott Reed
Cheng-Yang Fu
Alexander Berg

title: Ssd: Single shot multibox detector

authors:
Ze Liu
Yutong Lin
Cao Yue
Han Hu
Yixuan Wei
Zheng Zhang
Stephen Lin
Guo Baining

title: Swin transformer: Hierarchical vision transformer using shifted windows

venue: In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Yi Luo
Nima Mesgarani

title: A sequence matching network for polyphonic sound event localization and detection

venue: In EEE/ACM Transactions on Audio, Speech and Language Processing,

title: Group invariant scattering

venue: In Communications in Pure and Applied Mathematics,

authors:
Ishan Misra
Abhinav Shrivastava
Abhinav Gupta
Martial Hebert

title: Cross-stitch networks for multi-task learning

venue: In IEEE Conference on Computer Vision and Pattern Recognition

authors:
Satish Mohan
Michael Lockwood
Michael L. Kramer
Douglas L. Jones

title: Localization of multiple acoustic sources with small arrays using a coherence test

venue: In The Journal of Acoustical Society of America,

authors:
Mahesh Kumar Nandwana
T. Hasan

title: Towards smart-cars that can listen: Abnormal acoustic event detection on the road

venue: In Interspeech,

authors:
Dimitri Palaz
Ronan Collobert
Mathew Magimai-Doss

title: Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks

authors:
Daniel Park
William Chan
Yu Zhang
Chung-Cheng Chiu
Barret Zoph
Ekin Cubuk
Quoc V. Le

title: Specaugment: A simple data augmentation method for automatic speech

venue: URL https://arxiv.org/abs/1904.08779

authors:
Archontis Politis
Sharath Adavanne
Tuomas Virtanen

title: A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection

venue: In Detection and Classification of Acoustic Scenes and Events Workshop (DCASE),

authors:
Mirco Ravanelli
Yoshua Bengio

title: Speaker recognition from raw waveform with sincnet

venue: IEEE Workshop on Spoken Language Technology (SLT),

authors:
Tara N. Sainath
Brian Kingsbury
Abdel-rahman Mohamed
Bhuvana Ramabhadran

title: Learning filter banks within a deep neural network framework

venue: In IEEE Workshop on Automatic Speech Recognition and Understanding,

authors:
Tara N. Sainath
Ron J. Weiss
Andrew Senior
Kevin W. Wilson
Oriol Vinyals

title: Learning the speech front-end with raw waveform cldnns

authors:
Robin Scheibler
Eric Bezzam
Dokmanić Ivan

title: Pyroomacoustics: A python package for audio room simulations and array processing algorithms

authors:
Steffen Schneider
Alexei Baevski
Ronan Collobert
Michael Auli

title: wav2vec: Unsupervised pre-training for speech recognition

title: mallat: A wavelet tour of signal processing, 2nd edition

venue: Computer music journal,

authors:
Harshavardhan Sundar
Weiran Wang
Ming Sun
Chao Wang

title: Raw waveform based end-toend deep convolutional network for spatial localization of multiple acoustic sources

authors:
Tho Nguyen Thi Ngoc
Ngoc Khanh Nguyen
Huy Phan
Lam Pham
Kenneth Ooi
Douglas Jones
Woon-Seng Gan

title: A general network architecture for sound event localization and detection using transfer learning and recurrent neural network

authors:
T.N. Tho Nguyen
D.L. Jones
W. Gan

title: Affine structure from sound

venue: In Advances in Neural Information Processing Systems (NeurIPS),

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jacob Uszkoreit
Llion Jones Jone
Aidan N. Gomez
Lukasz Kaiser

title: Attention is all you need

authors:
Juan Manuel Vera-Diaz
Daniel Pizarro
Javier Macias-Guarasa

title: Towards end-to-end acoustic localization using deep learning: From audio signals to source position coordinates

authors:
Qiang Wang
Li Zhang
Luca Bertinetto
Weiming Hu
Philip H.S. Torr

title: Fast online object tracking and segmentation: A unifying approach

authors:
Qing Wang
Huaxin Wu
Zijun Jing
Feng Ma
Yi Fang
Yuxuan Wang
Tairan Chen
Jia Pan
Jun Du
Chin-Hui Lee

title: The ustc-iflytek system for sound event localization and detection of dcase2020 challenge

venue: In Tech. Report of SELD Decase2020 Challenge,

authors:
Bo Yang
Jianan Wang
Ronald Clark
Qingyong Hu
Sen Wang
Andrew Markham
Niki Trigoni

title: Learning object bounding boxes for 3d instance segmentation on point clouds

venue: In Advances in Neural Information Processing Systems,

authors:
Neil Zeghidour
Nicolas Usunier
Iasonas Kokkinos
Thomas Schatz
Gabriel Synnaeve
Emmanuel Dupoux

title: Learning filterbanks from raw speech for phone recognition

authors:
Neil Zeghidour
Olivier Teboul
Félix de Chaumont Quitry
Marco Tagliasacchi

title: Leaf: A learnable frontend for audio classification

venue: International Conference on Learning Representations (ICLR),

heading: 1 INTRODUCTION

text:
The fundamental task for an agent to perceive and interact with the 3D environment is to know the location and semantic identity of its nearby objects. The location includes spatial location that is either stationary or moving, temporal location like start time and end time. Vision-based such environment perception has received large attention in the past decade and we have witnessed huge progress in tasks such as object detection(Liu et al., 2016; Lin et al., 2014; Yang et al., 2019), classification(He et al., 2016a) and tracking(Wang et al., 2019). Nevertheless, the sound-based counterpart research has far lagged behind, despite all the fascinating properties sound signal exhibits. For example, sound is ubiquitous and insensitive to ambient illumination change, it has no field-of-view (FoV) constraints and is capable of circumventing physical barriers to perceive scene beyond line-of-sight. As a complementary sensing approach to vision, sound-based environment perception is of vital importance for acoustic scene understanding. A typical example is the sound source detection (answer where is it, when does it happen and what is it), given the recorded multi-channel sound waveforms.
To detect sound sources, we often deploy a spatially-configured microphone array to record an acoustic environment. Unlike camera or LiDAR scanner that directly captures RGB image or measures the range distance, sound waveform itself is a highly compressed one-dimensional points
with high sampling rate, all sound source signals are compressed and mixed into the one-dimensional data format. Since different sound sources have different frequency property, it is essential convert waveform into time-frequency representation so that frequencies hidden in the waveform are explicitly split out. This is often achieved by projecting the raw waveform into various orthogonal frequency basis. At the same time, a sound source’s spatial location clue lies in inter-channel difference of recorded waveforms (i.e. phase difference in this work). It is essential to design a neural network that jointly encodes mono-channel time-frequency representation and inter-channel phase difference from the raw waveforms in a unified, parameter-frugal and computation-efficient manner. The learned representation should have elegant resolution in both time, frequency and space domain so that sound sources can be precisely detected.
However, learning such representation is a tough task. Challenges derive from both theoretical side and practical side. According to Uncertainty Principle, we cannot achieve the optimal resolution in time and frequency domain at the same time, but instead keep a trade-off between them. Traditional hand-engineered sound feature(Davis & Mermelstein, 1980; Cao et al., 2021; Tho Nguyen et al., 2020; Brandstein & Silverman, 1997) and some recently proposed learnable filter bank(Ravanelli & Bengio, 2018; Zeghidour et al., 2018) empirically set the same length for all filters, resulting in human-biased, unadjustable time-frequency resolution map. Some other work(Zeghidour et al., 2021) correlates filter frequency response and filter length by initializing in mel-scale, but it is not scalable nor stable because the final initialization depends on the filter number. Moreover, all existing methods process raw waveform with one-scale filter bank, we argue that the one-scale sound perception easily leads to incomprehensive sound sources sensing, especially when sound sources have different frequency property or undergo various spatial motion.
In this paper, we first give comprehensive theoretical analysis on the filter bank impact on its extracted feature’s time and frequency resolution. Based on the analysis, we propose a simple yet effective synperiodic filter bank construction strategy in which each filter’s frequency response and length are correlated by rotating periodicity such that each filter’s length is inversely proportional to its frequency resolution. The synperiodic filter bank thus internally maintains a better time-frequency resolution trade-off than traditional fixed-length filter bank. Coupling the filter length with its frequency response helps us to reduce human intervention in filter bank design. By simply alternating the periodicity term, we further construct a group of synperiodic filter banks, with which we achieve multi-scale perception in time domain. At the same time, by applying a synperiodic filter bank to process one raw waveform as well as its consecutively-downsampled versions, we achieve multi-scale perception in frequency domain. The multi-scale perception in both time and frequency domain of synperiodic filter bank enables the neural network to dynamically learn better representation for sound source detection in a data-driven way. It is worth noting that synperiodic filter bank parameter number is just linear to filter number (adds up to less than 1% of the whole parameters) and it can be efficiently implemented as a 1D convolution operator.
Following the aforementioned learnable front-end, we add a Transformer-like backbone network with two paralleling branches with intermediate soft-parameter sharing to learn sound source’s semantic and spatial location related representation both jointly and separately. Experiment on both direction-of-arrival (DoA) task and physical location estimation task shows that our proposed framework outperforms comparing methods by a large margin. Replacing existing method’s head with our proposed synperiodic filter bank also improvdes the performance.

heading: 2 RELATED WORK

text:
Sound signal processing has been thoroughly studied in traditional digital signal processing area. The preliminary step of sound signal processing is usually to convert raw waveform into 2D timefrequency representation. There are two main realms: Fourier transform based and Wavelet based transform(Sturm, 2007). Traditional sound feature design are motivated influenced by human-auditory system. For example, they often convert frequency bins into mel-scale to imitate human hearing system, like MFCC(Davis & Mermelstein, 1980), LogMel(Cao et al., 2021; Grondin et al., 2019), the filter length is empirically chosen and often a windowing is added to avoid spectrum leakage. For inter-channel phase difference encoding, it is often recommended to encode in frequency domain due to the less-computation advantage. Typical phase difference features include GCC-Phat(Brandstein & Silverman, 1997) and intensity vector(Cao et al., 2021).
Sound source detection has been previously treated as sound structure(Thrun, 2006) estimation, sound object detection(He et al., 2021) and sound event detection and localization (SELD) problem(Adavanne et al., 2018; Thi Ngoc et al., 2021). It involves jointly identifying a sound source’s semantic label and predicting its spatial location, the two sub-tasks have been thoroughly studied separately in acoustics(Nandwana & Hasan, 2016; Mohan et al., 2008; Sundar et al., 2020; Vera-Diaz et al., 2018) and computer vision community(He et al., 2016a). T. Kim et al.(Kim et al., 2019) provided a review and discussion for raw-audio based event classification. Benefiting from the success of the traditional hand-engineered sound feature and the large availability of mature image-based deep neural networks(He et al., 2016a), most work(Grondin et al., 2019; Cao et al., 2021; Grondin et al., 2019; Adavanne et al., 2018; Thi Ngoc et al., 2021) tackle the task by first extracting hand-engineered sound feature and then feeding them to mature image-based neural networks. This workflow is straightforward and often guarantees reasonably good results but it is not end-to-end trainable and heavily depend on image side which may not be optimal for sound processing. At the same time, some work(He et al., 2021; Adavanne et al., 2018; Tho Nguyen et al., 2020) have simplified the problem by assuming no two sound sources of the same semantic label but different spatial location happen at the same time. This assumption avoids semantic label and spatial location association issue but may not reflect real scenarios.
In recent years, a bunch of work tried to design neural work to directly learn from raw sound waveform, ranging from the earlier methods that directly apply stacked layers to process raw waveform(Schneider et al., 2019; Palaz et al., 2013; Jaitly & Hinton, 2011; Sainath et al., 2013) to the recent frequencysensitive filter bank learning methods(Zeghidour et al., 2021; Ravanelli & Bengio, 2018; Zeghidour et al., 2018; He et al., 2021; Hoshen et al., 2015; Sainath et al., 2015; Luo & Mesgarani, 2019). The filter bank parameter is initialized in either mel-scale(Ravanelli & Bengio, 2018; Zeghidour et al., 2021; He et al., 2021; Zeghidour et al., 2018) or as Gammatone filter(Hoshen et al., 2015; Sainath et al., 2015). SoundDet(He et al., 2021) is the first work to directly learn from multi-channel raw waveforms to detect sound sources. It designs MaxCorr filter bank to directly convolve with multi-channel raw waveforms to learn phase difference aware features.
Multi-scale representation has a rich history in computer vision community ((Liu et al., 2016; 2021; Lin et al., 2016; He et al., 2016b)), in which multi-scale representation strategy has been proposed to accommodate large object scale variation. For example, SSD(Liu et al., 2016) combines feature maps of various scale to cover potential objects of various sizes. In addition to spatial multi-scale representation, feature space multi-scale representation has been explored as well. For example, He et. al.(He et al., 2016b) incorporate features arising from intermediate layers to boost attribute recognition. In sound signal processing, Stéphane Mallat(Bruna & Mallat, 2013; Mallat, 2012) proposed wavelet scattering to obtain multi-scale sound representation by iteratively treating the proceeding processed sound waveform as new virtual waveform for further process.

heading: 3 SOUND SOURCE DETECTION FROM MULTI-CHANNEL RAW WAVEFORMS

heading: 3.1 SOUND SOURCE DETECTION PROBLEM DEFINITION

text:
We use a spatially-deployed microphone array, like four closely-bounded microphones in a circular coplanar configuration, to record an acoustic environment where various sound sources undergo independent and unconstrained spatial motion. Recorded multi-channel raw waveforms are represented as W “ txiptqu4i“1, each waveform is 1-d vector of size T sampled at a fixed sampling rate. We denote K sound sources by S “ tsk “ ptks , tke , spk, slkquKk“1, one single sound source corresponds to a start time ts, end time te, stationary or moving spatial motion sp and a semantic identity label sl. Our goal is to learn a representation P that is representative enough to detect sound sources both spatially and semantically. P is a temporal framewise representation of shape T1 ˆN . We propose to learn P directly from the raw waveforms via a deep neural network F parameterized by θ,
PT1ˆN “ FpWTˆ4|θq, T1 ! T (1)
A learned elegant representation should have large resolution in both time, frequency and space domain and exhibit strong capability at addressing three intrinsic challenges: Polyphonicity, sound sources are mutually independent, they can freely overlap temporally; Time-scale variance, the
temporal duration of different sound sources varies enormously, from seconds to minutes long; Datasize, sound waveform is sampled with large sampling rate, resulting in large data size.
Sound source presence clue lies in mono-channel time-frequency representation and its spatial motion clue lies in inter-channel interaction, like phase difference. Our proposed synperiodic filter bank group jointly encodes time-frequency representation and phase difference from raw waveforms in a unified way. More detail about phase difference encoding is given in the Appendix A.1.

heading: 3.2 MULTI-SCALE SYNPERIODIC FILTER BANK

text:
We denote the general filter bank of M filters by F “ tFipwi, σi, liq “ φpwiq ¨ ωpσiquMi“1. Each single filter Fi is created by multiplying the frequency-selective filter φpwiq of frequency response wi (i.e. sinusoidal basis or band-pass basis(Ravanelli & Bengio, 2018)) with a windowing function ωpσiq. The windowing function has locality property which means it is just active within a local region controlled by σi (i.e. Gaussian kernel window). Then li points are cropped around the active region and treated as a frequency-selective filter (usually li ą σi). Existing filter bank differs in their way of parameter setting, including filters’ frequency response distribution, filter length length and windowing function. In classic hand-engineered features such as short-time Fourier transform (STFT) and LogMel feature, all these parameters are empirically chosen. Contrary to STFT, wavelet transform(Sturm, 2007) inversely correlates window length with frequency response so that the active region of the filter is internally decided by its frequency response.
Time-Frequency Dynamic Resolution Frequency resolution indicates the ability of discerning two adjacent frequency bins, time resolution corresponds to the capability of precisely localizing a sound source in time domain. The resolution in time and frequency domain jointly influence spatial localization precision because we encode spatial location on top of on pre-extracted time-frequency representation. According to Uncertainty Principle, however, the frequency resolution ∆f and time resolution ∆t satisfies ∆f ¨∆t ě C (C is a constant). It means we cannot get optimal resolution in both time and frequency domain at the same time, but rather keep a trade-off between them: increasing the resolution in one domain inevitably sacrifices the resolution in the other domain.
Traditional hand-engineered sound feature’s time-frequency resolution map is fixed because all filter bank construction relevant parameters are empirically chosen. Therefore, its time-frequency resolution map is evenly divided across both the time and frequency domain (see Fig.1 second last figure). Relaxing the frequency response and windowing length as trainable(Zeghidour et al., 2021) helps to achieve a better time-frequency resolution trade-off. The improvement is, however, limited because it uses one filter bank of the same filter length to process all sound signals (or perceive at one-scale). We categorize these traditional filter bank as syndistance filter bank to emphasize their equal temporal length property across all frequency responses.
In this paper, we rethink filter bank design and propose a synperiodic filter bank construction strategy. Our motivation is based on the fact that high-frequency sound signal can be adequately detected
with a short filter because its temporal length of one period is small. Conversely, lower frequency sound signals require wider filters. It thus shows using the same window size for all filters across the frequency range is not a good choice, a more desirable way is to use narrower window for filters with higher frequency responses. Under this guidance, we propose a synperiodic filter bank construction strategy. To better understand the main difference between syndistance filter bank and our proposed synperiodic filter bank, we can visualize them in complex-valued plane (see Fig.1, left-most), in which a filter is a complex exponential rotating in the complex plane counter-clockwisely, the rotating speed corresponds to the frequency it carries. In the complex-valued plane, all syndistance filters rotate to the same distance. Synperiodic filters, however, rotate to a predefined periodicity ρ, naturally resulting in narrow window for high-frequency filters and wide window for low-frequency filters.
Synperiodic filter bank lends us three advantages: it first avoids us setting window length for each filter separately, which is quite empirical and random; second, the constructed filter bank by design maintains a good time-frequency resolution trade-off; third, by simply varying the periodicity term ρ, we can easily obtain a group of synperiodic filter bank to process the raw waveform in multi-scale manner (as we will present in below). Our synperiodic filter bank construction strategy shares similar idea with Wavelet transform(Sturm (2007)) where it adopts a time shift and “squeezing ratio” to create achieve multi-scale perception, the difference is that we omit the time shift and instantiate the squeezing ratio with our proposed synperiodicity strategy. Moreover, synperiodic filter bank is multi-scale in time and frequency domain and seamlessly works well with convolution operation, and it is end-to-end trainable. Specifically, our proposed synperiodic filter bank can be represented as,
Fρsynp “ tfipwi, ρ, liq “ φpwiq ¨ ωpwi, ρquMi“1, where ωpσiq “ ωpwi, ρq (2)
where we can see that the windowing function just depends on the center frequency wi and the periodicity term ρ. By varying the periodicity term ρ, we can obtain a group of synperiodic filter bank Fsynp “ tFρ1 ,Fρ2 , ¨ ¨ ¨ ,Fρnu, each group differs in its window size. The comparison between syndistance and synperiodic filter bank is shown in Fig. 1.
There are many ways to instantiate ωpwi, ρq, as long as we guarantee the window length gradually reduces as the the frequency response increases. The simplest choice is to treat ωpwi, ρq as a constant, but we find it either results in too wide window for low-frequency filters or too narrow window for high-frequency filters. To mitigate this dilemma, we use logarithmic windowing function,
ωpwi, ρq “ 27 ¨ log10pwiq ´ ρ, ρ “ t´6,´11,´16u (3)
We set ρ as r´6,´11,´16s respectively to construct three synperiodic filter banks. The design of this window function is motivated by mel-scale frequency initialization strategy. By roughly setting a filter’s bank width to be equal to the distance between its preceding and next frequency location in frequency domain, converting to time domain we can roughly get a logarithmic scale frequencyperiodicity relationship (see Fig.4 in Appendix). In our implementation, synperiodic filter is created by multiplying a sinusoidal basis with learnable frequency response initialized in mel-scale by a Gaussian window with learnable width initialized through the windowing function by Eqn.3. Please note that each synperiodic filter bank group is initialized with independent learnable frequencies and window length, they are independently updated during training stage.

heading: 3.3 MULTI-SCALE LEARNING IN TIME AND FREQUENCY DOMAIN

text:
We use the previously constructed synperiodic filter bank group to convolve with mono-channel sound waveform with the same step size and padding strategy, resulting in the same size output for each single synperiodic filter bank. Since different filter group has different window size, we naturally achieve multi-scale learning in time domain. It maximally avoids us empirically selecting one window scale ρ which might not be optimal, but instead uses a group of filter bank to enforce the neural network to strike a better time-frequency resolution trade-off in a data-driven way.
We further propose a strategy to enable multi-scale learning in frequency domain. The strategy is hierarchical: given a raw sound waveform with sampling frequency FS , synperiodic filter bank’s frequency is initialized within the range r0, FS2 s under Nyquist sampling theorem. If we downsample the sound waveform by a factor of 2, the resulting waveform can be processed by the lower-half
filters in each group whose frequency response lies in r0, FS4 s. This process that 2x-downsampling the waveform further process the downsampled waveform with filters with lower-half frequency response can potentially iterate a couple of times (in our case three times), resulting in multi-scale perception in frequency domain. The figurative illustration is shown by Fig. 2. Multi-scale learning in frequency domain brings us two extra benefits: 1) from data augmentation perspective, hierarchical 2x downsampling inevitably creates more waveforms, which matters a lot when labelling audio data is technically hard and time-consuming. 2) from the causality perspective, the adjacent 2xdownsampling strategy leads to dilated convolution for lower-frequency filters, because applying a filter to convolve with a downsampled waveform equals to convolve on the original waveform with dilated convolution (skip-2 connection). The resulting wider field of view (FoV) for lower frequency filters enables to learn better sound causality along the time axis (see Appendix A.4). In sum, by using learnable synperiodic filter bank group to process the raw waveform in multi-scale manner, we achieve a dynamic time-frequency resolution map that naturally maintains a better time-frequency resolution map fitting for sound source detection in a data-driven way.
Computational Analysis Synperiodic filter bank group introduces very few parameters (less than 1%) because they are parameterized filters. The trainable parameter number increases linearly w.r.t. synperiodic filter bank number. Their convolution with raw waveforms can also be efficiently implemented with 1D convolution.

heading: 3.4 TANSFORMER-LIKE BACKBONE WITH TWO SOFT-STITCHING BRANCHES

text: Synperiodic filter bank group generates feature representation of shape rT0, N0, Cs (T0 is much smaller than the T but is still larger than T1. N0 equals to the filter number and we use 256, C is channel number). Jointly learning framewise sound source semantic label and spatial location representation is a multi-task problem(Kendall et al., 2018; Misra et al., 2016). We propose a Transformer-like backbone with two paralleling and identical branches to learn each sub-task separately. To enforce information communication, we add a layerwise information exchange module: for the intermediate semantic label feature f is and spatial location feature f i g learned by the i-th block, a learnable weight Wi is introduced to linearly combine them together to get an updated f is and f i g before feeding them to the next layer, rf is, f igs “ Wi ¨ rf is, f igs. In practice, the Transformer-like backbone gradually reduces the learned feature’s temporal length but increases the channel dimension accordingly. Finally we can get the final sound source representation of shape rT1, N s. On top of the representation, we add trackwise permutation-invariant training (PIT) strategy(Cao et al., 2021) to trains the whole neural network in an end-to-end manner. The permutation invariant training strategy helps to address the polyphonicity challenge because it encapsulates a potential sound source’s semantic label and spatial location within a track, so it won’t be influenced by other sound sources.

heading: 4 EXPERIMENTS

text:
We conduct experiments on two tasks: direction of arrival (DoA) and physical location estimation. DoA estimation requires to decide arrival angle of sound source to microphone array. Physical location
directly localizes a sound source’s physical location (i.e. px, y, zq coordinates). For DoA estimation task, we use DCASE2020 sound event detection and localization dataset(Politis et al., 2020). It contains 14 sound sources with azimuth range r´180˝, 180˝s and elevation range r´45˝, 45˝s. Two recording formats are available: FOA and MIC-array (refer to Appendix for detailed discussion). For more details about this dataset, please refer to (Politis et al., 2020). For physical location estimation, we use pyroomacoustics(Scheibler et al., 2018) simulator to simulate the dataset.

heading: 4.1 DIRECTION OF ARRIVAL ESTIMATION

text:
We adopt two evaluation metrics: segment-based and event-based metric. Segment-based metric is a widely adopted evaluation metric(Adavanne et al., 2018; Cao et al., 2021), it couples semantic label and spatial location together: a semantic-correctly detected sound source needs to be spatially close enough to its ground truth location in order to be regarded as a true positive detection. Event-based based metric is newly proposed by(He et al., 2021) to comprehensively evaluate under different confidence scores. Like object detection from images(Lin et al., 2014), it computes mean average precision (mAP) and mean average recall (mAR) score.
We call our framework SoundSynp and compare it with four most recent methods: SELDNet(Adavanne et al., 2018), EIN(Cao et al., 2021), SoundDet(He et al., 2021) and Utsc-Iflytek(Wang et al., 2020). SELDNet is the baseline model and it jointly trains sound source’s semantic label and spatial location with a convolutional recurrent neural network (CRNN)(Chung et al., 2014). EIN(Cao et al., 2021) is a very recent work. It adopts multi-heads self-attention(Vaswani et al., 2017) to model temporal dependency and trackwise permutation-invariant training to train the model. SoundDet(He et al., 2021) directly learns from raw waveform with MaxCorr kernels, followed by an encoder-decoder neural network. Utsc-Iflytek(Wang et al., 2020) is ranked first in DECASE2020 challenge leaderboard1, it combines MIC and FOA features and ensembles different models like ResNet(He et al., 2016a) and Xception(Chollet, 2017) to give final prediction.
Implementation Detail The network architecture is shown in table 4 in Appendix material. To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets. The raw waveform is first normalized to r´1, 1s before feeding the neural network. We adopt Adam optimizer(Kingma & Ba, 2015) with an initial learning rate 0.0002 in the first 100 epochs and 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classification head and regression head is 1 : 2. During training, data
1see this link for leaderboard report.
augmentation method SpecAugment(Park et al., 2019) is applied. For DoA task, we regress direction of arrival angle in Cartesian coordinates rx, y, zs. In synperiodic filter bank groups, the filter length is 1025, each group’s filter number is 256 and the step size is 600. Particularly, we have observed the initialized learnable synperiodic filter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable. The final learned parameters are close to their initialization, one group’s learned parameters (like frequency response and filter window length) is different from those learned by other groups, although they are initialized as the same in the start. We train each model five times independently and report the average score. The standard deviation is within 0.04 (for recall) and 0.2˝ for angle, 0.003 for mAP and mAR.
The result is given in Table1, from which we see that SoundSynp achieves the best performance over all comparing methods by a large margin, under both segment-based and event-based metrics. Both SELDNet, EIN and UTSC-Iflytek use pre-extracted hand-engineered sound features, such as Logmel, GCC-Phat and Intensity Vector. SoundDet(He et al., 2021) and SoundSynp are the only two methods that directly learn from raw waveforms. At the same time, SoundSynp obtains better performance on FOA than MIC format, the same phenomena has been observed by all other methods. It thus shows FOA better fits for sound source detection than MIC. It is worth noting that Utsc-Iflytek(Wang et al., 2020) ensembles different powerful image-based 2D models to detect sound sources. However, our proposed SoundSynp still outperforms Utsc-Iflytek by a large margin. We don’t report of mAP/mAR value for Utsc-Iflytek because it is a complex system and no detail about their system is available.
Ablation Study To disentangle the individual contribution of each part of our SoundSynp framework to the whole performance improvement, we conduct three ablation studies. First, the individual contribution of Synperiodic filter bank. We replace hand-engineered sound feature head of SELDNet, EIN and learnable MaxCorr filter bank with our proposed Synperiodic filter bank to test their corresponding performance. It helps to remove the influence of the backbone neural network of different models and thus helps to get direct comparison of synperiodic filter bank with other front-end. Second, we replace SoundSynp’s synperiodic filter bank group with widely-used MFCC(Davis & Mermelstein, 1980) and Log-Mel that is used by SELDNet(Adavanne et al., 2018) and EIN(Cao et al., 2021), respectively. It disentangles synperiodic filter bank group with Transformer-like backbone, thus helps to figure out if the performance gain is simply brought by backbone network. Third, internally, we test five synperiodic variants: (1) synperiodic filter bank with just multiscale perception in frequency domain (SoundSynp MSFreq), (2) just multi-scale perception in time domain (SoundSynp MSTime), (3) Synperiodic filter bank with frequency responses linearly initialized in Nyquist frequency range (SoundSynp Linear, compare with our mel-scale initialization), (4) just one synperiodic filter bank without multi-scale perception neither in time nor frequency domain (SoundSynp SingleScale). (5) synperiodic filter bank with rectangular band-pass frequency response initialization (SoundSynp Sinc), like SincNet(Ravanelli & Bengio, 2018) does. The internal comparison helps us to figure out the necessity of each part of synperiodic filter bank design.
The ablation study result is shown in Table 2. We can observe that: First, using synperiodic filter bank as a replacement of existing filter bank can help to improve the corresponding performance. Second, replacing SoundSynp’s synperiodic filter bank with classic hand-engineered features inevitably reduces the performance under all evaluation metrics, its thus shows learning from pre-extracted fixed and single-scale sound representation leads to inferior performance than our proposed multiscale synperiodic filter bank. Third, the absence of multi-scale perception in either frequency domain or time domain inevitably reduces the performance. We find sound source semantic label detection suffers more in single-scale perception in time domain than in frequency domain (see better performance on ER20˝ , and F20˝ score), which shows frequency domain multi-scale perception is vital for semantic label estimation. Similarly, we can observe that multi-scale perception in time domain is vital for sound source spatial location estimation (see better performance on LE and LR score). Linearly initialized filter bank frequency response reduces the performance, which shows assigning more filters to the lower frequency range is important. But this conclusion might be data-dependent because we find DCASE dataset contains many low-frequency sounds like burning fire and footsteps. Moreover, reducing the synperiodic filter bank group to one groups with just singlescale perception leads to comparably the worst performance, it thus shows multi-scale perception in both time and frequency domain is essential for DoA-based sound source detection. Lastly, SoundSynp Sinc leads to slightly inferior performance than our used mel-scale initialization strategy, it shows our proposed synperiodic filter bank is a general filter bank that can be adopted to other frequency sensitive filter bank.
One qualitative comparison is shown in Fig. 3. We can clearly see that SELDNet generates mixed prediction at different time steps and DoA locations. SoundDet and EIN give non-existing sound sources (orange color). When multiple sound sources happen at the same time (polyphonicity), SoundDet and EIN are easily failed to predict the right spatial location (discretized blue and red color). Our method predicts more spatially and temporally consistent sound sources by maximally keeping sound source’s continuity and consistency.

heading: 4.2 PHYSICAL LOCATION ESTIMATION

text: We use Pyroomacoustics simulator(Scheibler et al., 2018) to simulate a shoebox like room with r7m, 5m, 3ms size. Four microphones (24kHz sampling rate) are configured in a plane with 5cm mutual distance and put in the room center (r3.5m, 2.5m, 1.5ms). We choose three commonly heard sounds from BBC sound effect site2: cat meowing, baby talking and dog barking. All these seed sounds last less than 10s. During simulation, we put all seed sound in the plane z “ 1.5m (we constrain to a plane because we find relaxing z easily leads EIN model to get stuck in a local minima like r´1,´1,´1s), but with x randomly in r2m, 5ms and y randomly in r1m, 4ms, up to two sounds can happen at the same time. In sum, we have simulated 500 one-minute recordings (400 for train, 100 for test), with an average of four sounds recorded per recording. For physical location prediction, we regress the angle and range. The result is in Table 3. We do not report SELDNet result as we find it doesn’t converge during train. We can see from the table that SoundSynp produces the best performance than over EIN and SoundDet network. Replacing their front-end with synperiodic filter bank also improves the performance. It thus shows our proposed SoundSynp is capable of learning representation for physical-location based sound source detection.

heading: 4.3 DISCUSSION AND CONCLUSION

text:
We have proposed a novel learnable sound feature extractor front-end: synperiodic filter bank. It requires very few parameters and is computational efficient. It extracts feature in multi-scale manner in both time and frequency domain. Although it shows excellence in detecting sound sources, we believe it can be used as a general front-end to tackle other sound tasks, like speech processing and keyword spotting.
2see https://sound-effects.bbcrewind.co.uk/

heading: A APPENDIX

text:
A.1 SPATIAL LOCATION ENCODING IN FREQUENCY DOMAIN AND RECORDING FORMAT DISCUSSION
We discuss the detailed spatial location encoding for FOA and MIC sound waveform recording format. The spatial location encoding is based on 2D feature learned by the Gabor filter bank for each sound waveform channel, which can be represented as tFi “ pRi, Iiqu4i“1. Ri and Ii are real part and imaginary part feature of the i-th channel sound waveform, respectively.
FOA format is well-known as first-order Ambisonics (B-format). It contains four channels: omnidirectional, x-directional, y-directional and z-directional components, respectively. The instantaneous sound intensity vector is often used as spatial location (or phase difference) feature, which can be computed through the cross-spectrum between the omni-directional channel to the remaining x, y, zdirectional. As a result, we have obtained 3 channel spatial location encoding feature.
IVx “ F˚0 ¨ F1, IVy “ F˚0 ¨ F2, IVz “ F˚0 ¨ F3 (4)
where F˚0 indicates the conjugate of the omni-directional feature. The three cross-spectrum feature IVx, IVy and IVz are stacked together and further normalized before serving as the spatial encoding feature.
MIC format is well-known as tetrahedral microphone array. The four microphones are mounted in spherical coordinates with four distinct orientations. We treat the four microphones equally and compute the phase difference between any two microphones. Thus a total of six channels spatial location feature can be constructed. Specifically, we choose to compute GCC-PHAT(Brandstein & Silverman, 1997) like cross-spectrum feature. For any two channel m and n, we compute the angle between the real part and imaginary part of the cross-spectrum.
SL “ anglepF˚m ¨ Fnq, m ‰ n,m “ 1, 2, 3, 4;n “ 1, 2, 3, 4 (5)
SL indicates the spatial location feature computed by the sound waveform channel m and n. The anglep¨q equals to a frequency amplitude normalization operation, like the GCC-PHAT(Brandstein & Silverman, 1997) does. Please note that all the spatial location feature computation operations are differentiable so the whole neural network becomes end-to-end trainable.
A.2 NETWORK ARCHITECTURE AND TRAINING DETAILS
SoundSynp neural network architecture is given in Table 4. To train the neural network, we evenly divide the one minute long four-channel raw waveform into non-overlapping 4s short snippets. The raw waveform is first normalized to r´1, 1s before feeding the neural network. We adopt Adam optimizer(Kingma & Ba, 2015) with an initial learning rate 0.0002 in the first 100 epochs and 0.00007 in the following 50 epochs. Batchsize is 16. The loss combination weight between classification head and regression head is 1 : 2. During training, data augmentation method SpecAugment(Park et al., 2019) is applied. For DoA task, we regress direction of arrival angle in Cartesian coordinates rx, y, zs. In synperiodic filter bank groups, the filter length is 1025, each group’s filter number is 256 and the step size is 600. Particularly, we have observed the initialized learnable synperiodic filter banks update its parameters intensively during the very early several epochs, and then gradually becomes stable. The final learned parameters are close to their initialization, one group’s learned parameters (like frequency response and filter window length) is different from those learned by other groups, although they are initialized as the same in the start.
A.3 SYNPERIODIC FILTER BANK FREQUENCY-PERIODICITY RELATIONSHIP DETERMINATION
Mel-scale time-frequency representation has been widely used in both traditional sound feature like MFCC(Davis & Mermelstein, 1980), LogMel and learnable filter bank(Zeghidour et al., 2021). It initializes the filter bank in frequency domain, in which high-frequency filter has wider window length. We transform the filter bank into time domain and can naturally get a roughly logarithmic-scale frequency-periodicity relationship, in which narrower window width is associated with high-frequency
filters. We thus set ωpwi, ρq “ 27 ¨ log10pwiq ´ ρ. We plot our synperiodic filter bank window function and the mel-scale initialized windowing function in Fig. 4, it shows our proposed windowing function naturally approximates the mel-scale windowing function.
A.4 CAUSALITY IN MULTI-SCALE FREQUENCY DOMAIN PERCEPTION
Fig.5 shows how causality is achieved in multi-scale frequency domain perception.
A.5 COMPUTATION TIME REPORT
The computation time of all comparing methods is given in Table 5, from which we can see that our proposed SoundSynp framework requires between 2x-3x more computation time than SELDNet, EIN and SoundDet due to its large Transformer-like backbone. However, SoundSynp computation time is still within a controllable range.