DISTRIBUTED TRANSFER LEARNING FOR DEEP CONVOLUTIONAL NEURAL NETWORKS BY BASIC PROBABILITY ASSIGNMENT
Transfer learning is a popular practice in deep neural networks, but fine-tuning of a large number of parameters is a hard challenge due to the complex wiring of neurons between splitting layers and imbalance class distributions of original and transferred domains. Recent advances in evidence theory show that in an imbalance multiclass learning problem, optimizing of proper objective functions based on contingency tables prevents biases towards high-prior classes. Transfer learning usually deals with highly non-convex objectives and local minima in deep neural architectures. We propose a novel distributed transfer learning to tackle both optimization complexity and class-imbalance problem jointly. Our solution imposes separated greedy regularization to each individual convolutional filter to make single-filter neural networks such that the minority classes perform as the majority ones. Then, basic probability assignment from evidence theory boosts these distributed networks to improve the recognition performance on the target domains. Our experiments on several standard datasets confirm the consistent improvement as a result of our distributed transfer learning strategy.
Transfer learning is a popular practice in deep neural networks, but fine-tuning of a large number of parameters is a hard challenge due to the complex wiring of neurons between splitting layers and imbalance class distributions of original and transferred domains. Recent advances in evidence theory show that in an imbalance multiclass learning problem, optimizing of proper objective functions based on contingency tables prevents biases towards high-prior classes. Transfer learning usually deals with highly non-convex objectives and local minima in deep neural architectures. We propose a novel distributed transfer learning to tackle both optimization complexity and class-imbalance problem jointly. Our solution imposes separated greedy regularization to each individual convolutional filter to make single-filter neural networks such that the minority classes perform as the majority ones. Then, basic probability assignment from evidence theory boosts these distributed networks to improve the recognition performance on the target domains. Our experiments on several standard datasets confirm the consistent improvement as a result of our distributed transfer learning strategy.
1 INTRODUCTION
In supervised learning, many classification algorithms assume the same distribution for training and testing data. Consequently, change of distribution requires rebuilding of the statistical models which is not always practical because of the hardship of recollecting of training data or heavy learning process. One of the solutions is transfer learning that transfers the classification knowledge into a new domain Pan & Yang (2010). This aims at learning of highly-generalized models with different probability distributions across domains to learn novel domains without labeled data Wang & Schneider (2014) Zhang et al. (2013). Here, the main challenge is to reduce the shifts in data distribution between domains by algorithms that minimize the discriminant of the domains. It is worth mentioning that this could not get rid of domain-specific variations Long et al. (2016).
Transfer learning for deep neural networks has been proved highly beneficial to boost their overall performance. Deep learning practices usually require huge amount of labeled data to learn powerful models. The transfer learning enables adaptation to a different source with small training samples. On the other hand, deep neural networks practically learn intermediate features. They could provide better transfer among domains because some of them generalize well among various domains of knowledge Glorot et al. (2011). These transferable features generally underlies several probability distributions Oquab et al. (2014) which reduce the cross-domain discrepancy Yosinski et al. (2014).
The common observation among several deep architectures is that features learned in bottom layers are not that specific, but transiting towards top layers makes them tailored to a dataset or task. A recent study Yosinski et al. (2014) of the generality or specificity of deep layers for the sake of transfer learning reveals two difficulties which may affect the transfer of deep features. First, top layers get quite specialized to their original tasks and second, some optimization difficulties rise due to the splitting of the network between co-adapted layers. In spite of these negative effects, it
Algorithm 1 Basic Probability Assignment (BPA) Input: train/validation set X Output: basic probability assignment BPA(φ)
1: compute R(φ) and P(φ) (Eqs.1- 2) 2: calculate recall and precision assignments(Eq.3) 3: apply Dempster rule for accumulation (Eq.4)
is shown that transferred features not only perform better than random ones but also provide better initialization. This gives a boost to the generalization of deep neural networks as well.
In this paper, we propose a framework for distributed transfer learning in deep convolutional networks. This tries to alleviate the burden of splitting networks in the middle of fragile co-adapted layers. The intuition is that above difficulty relates to the complexity of deep architectures and also, class-imbalance in the transferred domain.
On the matter of network complexity, we argue that the splitting of layers leads to a hard optimization problem because of high complexity in the interconnections between neurons of co-adapted layers. It seems that transfer learning is not able to thoroughly reconstruct the original powerful wiring for the transferred domain. This is due to the size of network and large number of interconnections across neurons. To address this issue, we fine-tune the convolutional filters separately and hence, reduce the complexity of the non-convex optimization.
On the other hand, it seems that the class-imbalance problem rises form different distribution of data in original and transferred domains. This issue can be handled by cost-sensitive imbalanced classifications methods. By class-imbalance in transferred domain, we mean variable coverage of common classes in this domain and the ones from the original domain. It is probable that both original and transferred datasets have uniform distributions of data among their classes, but some classes in one domain may be fully or partly covered by the other domain. This results in imbalance class distribution in the transfer learning.
The determination of a probabilistic distribution from the confusion matrix is highly effective to produce a probability assignment which contributes to class-imbalance problems. This basic probability assignment can be either constructed from recognition, substitution and rejection rates Xu et al. (1992) or both precision and recall rates of each class Deng et al. (2016). The key point is harvesting of maximum possible prior knowledge provided by the confusion matrix to overcome the imbalance classification challenge.
Since the power of deep convolutional models come from mutual optimization of all parameters, we join the above distributed fine-tuned filters by a boosting scheme based on basic probability assignment. Our experiments confirm the functionality of our distributed strategy for deep transfer learning. The rest of paper is organized as follows. We present the formulation of our method in Section 2, report our experiments in Section 3 and conclude in Section 4.
2 FORMULATION
In general, a confusion matrix represents the class-based predictions against actual labels in form of a square matrix. Inspired by Dempster−Shafer theory, construction of basic probability assignment (BPA) Sentz & Ferson (2002) gives a vector which is independent of the number of class samples and sums up to one for each individual label. This basic probability assignment provides the ability to reflect the difference contributions of a classifier to each individual classes or combine the outcomes of multiple week classifiers.
2.1 BASIC PROBABILITY ASSIGNMENT
A raw two-dimensional confusion matrix indexed by predicted classes and actual labels provides some common measures of classification performance. They are accuracy (the proportion of the total number of predictions that were correct), precision (a measure of the accuracy provided that
a specific class has been predicted), recall (a measure of the ability of a prediction model to select instances of a certain class from a dataset) and F-score (the harmonic mean of precision and recall) Sammut & Webb (2011).
Suppose a set of train/validation samples X = {X1, . . . , X|X |} from C = {C1, . . . , C|C|} different classes are assigned to a label set L = {L1, . . . , L|L|} by a classifier (φ) such that |C| = |L|. If each element (nij) of the confusion matrix C(φ) is considered as the number of samples belonging to class Ci which assigned to label Lj , then we can define recall (rij) and precision (pij) ratios as follows Deng et al. (2016)
rij = nij∑|C| j=1 nij pij = nij∑|L| i=1 nij
(1)
It can be seen that the recall ratio is summed over the actual labels (rows) whilst the precision ratio is accumulated by the predicted classes (columns) of the confusion matrix C(φ). Now, we are able to define recall and precision matrices as
R(φ) = {rij} P(φ) = {pij}
for i ∈ [1 . . . |L|], j ∈ [1 . . . |C|] (2)
The basic probability assignments of these matrices contain recall and precision probability elements for each individual class Ci such that
mri = rii∑|C| j=1 rji mpi = pii∑|L| j=1 pij
(3)
These elements are synthesized to form the final probability assignments representing the recognition ability of classifier φ to each of the classes of set C
mi = mri ⊕mpi = mri ×mpi 1− ∑|C| i=1mri ×mpi (4)
Here, operator ⊕ is an orthogonal sum which is applied by Dempster rule of combination Sentz & Ferson (2002). The overall contribution of the classifier φ cab be presented as a probability assignment vector
BPA(φ) = {mi} for i ∈ [1 . . . |C|] (5)
It is worth mentioning that BPA(φ) should be computed by the train/validation set because we assume that the test set does not include actual labels. Besides, combination of different classes under vertical or horizontal categories is a common practice in visual classification. The benefit lies in the fact that bottom layers of deep convolutional architectures make better contribution to detect first and second order features that are usually of specific directions (vertical vs horizontal) rather than detailed distinguished patterns of the objects. This leads to a powerful hierarchical feature learning in the case that |C| |L|. In contrast, some classes can be divided to various sub-categories although they all get the same initial labels and hence this holds |C| |L| to take the advantage of top layers. In the above formulation, we do not merge or divide the original setup of the datasets under study (|C| = |L|) although it seems that our BPA-based approach is also able to boost the trained classifiers for each of the merge/divide scenarios.
2.2 DISTRIBUTED TRANSFER LEARNING
A general practice in transfer learning includes training of an original deep neural network on a dataset and then, fine-tuning of learned features for another dataset on a new target network. Bengio et al. (2012). The generality of selected features for both original and target domains is critical to the success of the transfer learning. For implementation, we train the original network and copy its bottom layers to form the target network. The top layers of the target network are initialized randomly and trained on the target dataset. We are able to employ backpropagation from top to bottom layers and fine-tune their parameters for the target task or freeze the copied originals and only update top target layers. This can be decided by size of the target dataset and number of parameters in the original layers. Fine-tuning of large networks for small dataset leads to overfitting but for small network or large dataset, performance will be improved Sermanet et al. (2013).
Based on our formulation for basic probability assignment (BPA) on Section 2.1, we are able to follow the above transfer learning procedure by learning of a classifier φ (SVM or Softmax) and computing BPA(φ) using Algorithm 1. Here, the learning means fine-tuning of target domain using the rained weights and biases of the original network. To implement this, we train the original fullyconnected layers by the features calculated by presenting target’s train set to convolutional layers of the same original network. We deploy this procedure for each of the available convolutional filters separately and compute the BPA of each individual single-filter network for train/validation sets. Then, we combine unary potentials of all the fine-tuned classifiers by employing BPA weights to come up with a unit set of class probabilities. Figure 1 provides an overview of conventional and distributed transfer learning processes.
Suppose that Ci is the predicted class for a test sample T provided by classifier φ. To revise the classification outcome by the BPA calculation, we multiply the test sample’s unary potentials U(T ) = {u1, . . . , u|C|} (probabilities of belonging to each class) by an assignment vector M(φ) = {1 − m1, . . . , 1 − m|C|} (contributions of the classifier φ to each class) and pick the maximum index as the revised predicted label
C(T ) = I ( arg max {u1 × (1−m1), . . . , u|C| × (1−m|C|)} ) (6)
This implies that if classifier φ performs well on class Ci (high mi), it is highly probable that C(T ) leans towards Ci. At the same time, other minority classes like Cj (low mj) have a chance to win if their unary potentials would be high enough (uj > ui). In contrast, if φ does poor classification on class Ci (low mi), the possibility of updating C(T ) to another class (Cj) with even worse unary potential (uj < ui) would be higher. Therefore, BPA shows quite successful in handling imbalance data distribution among classes.
Algorithm 2 Distributed Transfer Learning Input: train/validation set X , test sample T , set of week classifiers F Output: predicted class CF (T )
for i = 1 to |C| do for j = 1 to |F| do
1: compute mij ∈ BPA(F) (Alg.1) 2: calculate unary potential uij ∈ UF (T)
end for end for
3: predict boosted output CF (T ) (Eq.8) 4: employ error backpropagation for fine-tuning
As described in Section 1, employing probability assignment addresses the class-imbalance problem but does not reduce the complexity of optimization because of the fact that both forward learning and error backpropagation are applied to all the model parameters. To break this non-convex optimization, we introduce our distributed transfer learning strategy. For implementation, we replace the mutual learning of all the parameters with learning of each individual convolutional filter in a separate classifier fed by the bottom original layer. It means that we train a set of week single-filter classifiers F = {φ1, . . . , φ|F|} which |F| equals the number of convolutional filters in the deep neural architecture.we follow the recipe of single classifier in Equation 5 but extend it to redefine
BPA(F) = {mij} for i ∈ [1 . . . |C|], j ∈ [1 . . . |F|] (7)
such that mij is the probability assignment of class Ci to week single-filter classifier φj . To come up with class of the test sample T , we update the Equation 6 as follows
CF (T ) = I ( arg max { u1j × (1−m1j)∑F
j=1 u1j × (1−m1j) , . . . , uij × (1−m|C|j)∑F j=1 u|C|j × (1−m|C|j)
} )
(8)
Here, uij is the unary potential of class Ci determined by the week single-filter classifier φj . Building on the above formulations, we are able to distribute the transfer learning among convolutional filters and join them later to implement a better fine-tuning for the target deep convolutional network according to the Algorithm 2.
3 EXPERIMENTS
We conduct our experiments on MNIST, CIFAR and Street View House Numbers (SVHN) datasets. The MNIST dataset LeCun et al. (1998) contains 60, 000 training examples and 10, 000 test samples normalized to 20 × 20, centered by center of mass in 28 × 28 and sheared by horizontally shifting such that the principal axis is vertical. The foreground pixels were set to one and the background to zero. The CIFAR dataset Krizhevsky & Hinton (2009) includes two subsets. CIFAR-10 consists of 10 classes of objects with 6, 000 images per class. The classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. It was divided to 5, 000 randomly selected images per class as training set and the rest, as testing samples. The second subset is called CIFAR-100 having 600 images in each of 100 classes. These classes also come in 20 super-classes of five class each. The SVHN dataset Netzer et al. (2011) was extracted from a large number of Google Street View images by automated algorithms and the Amazon Mechanical Turk (AMT) framework. It consists of over 600, 000 labeled characters in full numbers and MNIST-like cropped digits in 32×32. Three subsets are available containing 73, 257 digits for training, 26, 032 for testing and 531, 131 extra samples.
We consider two different scenarios to evaluate the performance of our distributed transfer learning algorithm. In the first experiment, we try to observe the performance of fine-tuning for pairs
of datasets with close data distributions or number of classes. We select MNIST & SVHN and CIFAR-10 & CIFAR-100 as original-target domains and report the transfer learning results in form of train-test errors. In the second experiment, we apply transfer learning for pairs of datasets with far data/class setups which are MNIST & CIFAR-10 and SVHN & CIFAR-100. In this experiment, we arrange the datasets to examine the effect of dissimilar distributions rather than overfitting.
Before moving forward to discuss the experiments, we report the baseline train-test errors for the datasets in Table 1. These results are produced by the deep learning library provided by the Oxford Visual Geometry Group Vedaldi & Fulkerson (2008).
3.1 EXPERIMENT 1
Table 2 shows the performance of conventional and distributed transfer learnings for the first scenario. The first values before dash correspond to the training errors (left) and the second ones present the testing errors (right).
In this experiment, we target two pairs of datasets (original-target domains) which contain similar data and perform number/object recognition tasks. We report the results for both conventional and our distributed transfer learning methods. By conventional Bengio et al. (2012), we mean training the original dataset and fine-tuning of the target one. With distributed, we aim at training the original dataset but employing the basic probability assignment for the transfer learning.
It can be seen that the results for the conventional transfer learning follows our argument on size of network and number of model parameters Sermanet et al. (2013). Compared to Table 1, MNIST does a poor job on transferring of SVHN due to the overfitting of SVHN over MNIST network. In contrast, SVHN perform quite well on transferring MNIST.
On the other hand, transferring of SVHN from MNIST does not overfit when our distributed transfer learning is employed. In both settings of original-target domains, our distributed strategy outperforms the conventional transfer learning approach.
The experiment on CIFAR pair exposes more interesting results due to the fact that both datasets have the same number of samples but completely different distributions among the classes. In practice, CIFAR-100 includes all the classes of CIFAR-10 but CIFAR-10 does not have any clue of the several classes of CIFAR-100. The conventional experiments show that CIFAR-10 transfers well on CIFAR-100 but it cannot perform transferring although the target network does not overfit.
All in all, the performance of our distributed transfer learning (bold values) is better than the conventional scheme and also, outperforms the baseline deep learning practices.
3.2 EXPERIMENT 2
In Table 3, we reports the results for both conventional and distributed transfer learnings on the second scenario. Here, we pair datasets such that the similarity of their data distributions and number of classes get minimized and they are originally trained for different tasks. It is obvious that our distributed transfer learning outperforms all the conventional results.
For the first setup, CIFAR-10 does a better transfer learning than MNSIT although the number of classes are the same. It seems that CIFAR-10 provides better generalization due to higher diversity among its classes. Here, our distributed algorithm performs better than the conventional process and,
targeting of MNIST on CIFAR-10 network gives close performance to the deep learning outcomes. The second setup leads to the overfitting of SVHN over CIFAR-100 network due to huge number of samples. The other outcome is the poor performance of transferring CIFAR-100 over SVHN network as a result of huge conceptual gap between original-target domains.
Our observations show that fine-tuning on training set and calculating BPA on validation, result in better generalization of the transferred model on testing set. On the other hand, computing of BPA on training plus validation sets gives higher performance in case of hugely different number of classes in original-target datasets. Since we employ BPA to address the class-imbalance problem, we reckon that it better captures the distribution of data by adjoining both train/validation sets especially when we intend to transfer few classes of original dataset to the larger number of classes in the target.
4 CONCLUSION
We introduce a novel transfer learning for deep convolutional networks that tackles the optimization complexity of a highly non-convex objective by breaking it to several distributed fine-tuning operations. This also resolves the imbalance class coverage between original-target domains by using basic probability assignment across several week single-filter classifiers. By the above boosting, the overall performance shows considerable improvement over conventional transfer learning scheme. We conduct several experiments on publicly available datasets and report the performance as traintest errors. The results confirm the advantage of our distributed strategy for the transfer learning.
Update: I thank the author for his comments! At this point, the paper is still not suitable for publication, so I'm leaving the rating untouched.
This paper proposes a transfer learning method addressing optimization complexity and class imbalance.
My main concerns are the following:
1. The paper is quite hard to read due to typos, unusual phrasing and loose use of terminology like “distributed”, “transfer learning” (meaning “fine-tuning”), “softmax” (meaning “fully-connected”), “deep learning” (meaning “base neural network”),  etc. I’m still not sure I got all the details of the actual algorithm right.
2. The captions to the figures and tables are not very informative – one has to jump back and forth through the paper to understand what the numbers/images mean.
3. From what I understand, the authors use “conventional transfer learning” to refer to fine-tuning of the fully-connected layers only (I’m judging by Figure 1). In this case, it’s essential to compare the proposed method with regimes when some of the convolutional layers are also updated. This comparison is not present in the paper.
Comments on the pre-review questions:
1. Question 1: If the paper only considers the case |C|==|L|, it would be better to reduce the notation clutter.
2. Question 2: It is still not clear what the authors mean by distributed transfer learning. Figure 1 is supposed to highlight the difference from the conventional approach (fine-tuning of the fully-connected layers; by the way, I don’t think, Softmax is a conventional term for fully-connected layers). From the diagram, it follows that the base CNN has the same number of convolutional filters at every layer and, in order to obtain a distributed ensemble, we need to connect (for some reason) filters with the same indices. This does not make a lot of sense to me but I’m probably misinterpreting the figure. Could the authors revise the diagram to make it clearer?
Overall, I think the paper needs significant refinement in order improve the clarity of presentation and thus cannot be accepted as it is now.