In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.
1 INTRODUCTION
Deep convolutional neural networks have continuously demonstrated their excellent performances on diverse computer vision problems. In image classification, the milestones of such networks can be roughly represented by LeNet (LeCun et al., 1989), AlexNet (Krizhevsky et al., 2012), VGG net (Simonyan & Zisserman, 2014), GoogLeNet (Szegedy et al., 2014), and ResNet (He et al., 2015), with networks becoming deeper and deeper. However, the architectures of these network are significantly altered and hence are not backward-compatible. Considering a life-long learning system, it is highly desired that the system is able to update itself from the original version established initially, and then evolve into a more powerful one, rather than re-learning a brand new one from scratch.
Network morphism (Wei et al., 2016) is an effective way towards such an ambitious goal. It can morph a well-trained network to a new one with the knowledge entirely inherited, and hence is able to update the original system to a compatible and more powerful one based on further training. Network morphism is also a performance booster and architecture explorer for convolutional neural networks, allowing us to quickly investigate new models with significantly less computational and human resources. However, the network morphism operations proposed in (Wei et al., 2016), including depth, width, and kernel size changes, are quite primitive and have been limited to the level of layer in a network. For practical applications where neural networks usually consist of dozens or even hundreds of layers, the morphing space would be too large for researchers to practically design the architectures of target morphed networks, when based on these primitive morphing operations only.
Different from previous work, we investigate in this research the network morphism from a higher level of viewpoint, and systematically study the central problem of network morphism on the module level, i.e., whether and how a convolutional layer can be morphed into an arbitrary module1, where a module refers to a single-source, single-sink acyclic subnet of a neural network. With this
†Tao Wei performed this work while being an intern at Microsoft Research Asia. 1Although network morphism generally does not impose constraints on the architecture of the child network,
in this work we limit the investigation to the expanding mode.
modularized network morphing, instead of morphing in the layer level where numerous variations exist in a deep neural network, we focus on the changes of basic modules of networks, and explore the morphing space in a more efficient way. The necessities for this study are two folds. First, we wish to explore the capability of the network morphism operations and obtain a theoretical upper bound for what we are able to do with this learning scheme. Second, modern state-of-the-art convolutional neural networks have been developed with modularized architectures (Szegedy et al., 2014; He et al., 2015), which stack the construction units following the same module design. It is highly desired that the morphing operations could be directly applied to these networks.
To study the morphing capability of network morphism and figure out the morphing process, we introduce a simplified graph-based representation for a module. Thus, the network morphing process can be formulated as a graph transformation process. In this representation, the module of a neural network is abstracted as a directed acyclic graph (DAG), with data blobs in the network represented as vertices and convolutional layers as edges. Furthermore, a vertex with more than one outdegree (or indegree) implicitly includes a split of multiple copies of blobs (or a joint of addition). Indeed, the proposed graph abstraction suffers from the problem of dimension compatibility of blobs, for different kernel filters may result in totally different blob dimensions. We solve this problem by extending the blob and filter dimensions from finite to infinite, and the convergence properties will also be carefully investigated.
Two atomic morphing operations are adopted as the basis for the proposed graph transformation, based on which a large family of modules can be transformed from a convolutional layer. This family of modules are called simple morphable modules in this work. A novel algorithm is proposed to identify the morphing steps by reducing the module into a single convolutional layer. For any module outside the simple morphable family, i.e., complex module, we first apply the same reduction process and reduce it to an irreducible module. A practical algorithm is then proposed to solve for the network morphism equation of the irreducible module. Therefore, we not only verify the morphability to an arbitrary module, but also provide a unified morphing solution. This demonstrates the generalization ability and thus practicality of this learning scheme.
Extensive experiments have been conducted based on ResNet (He et al., 2015) to show the effectiveness of the proposed morphing solution. With only 1.2x or less computation, the morphed network can achieve up to 25% relative performance improvement over the original ResNet. Such an improvement is significant in the sense that the morphed 20-layered network is able to achieve an error rate of 6.60% which is even better than a 110-layered ResNet (6.61%) on the CIFAR10 dataset, with only around 1/5 of the computational cost. It is also exciting that the morphed 56-layered network is able to achieve 5.37% error rate, which is even lower than those of ResNet-110 (6.61%) and ResNet-164 (5.46%). The effectiveness of the proposed learning scheme has also been verified on the CIFAR100 and ImageNet datasets.
2 RELATED WORK
Knowledge Transfer. Network morphism originated from knowledge transferring for convolutional neural networks. Early attempts were only able to transfer partial knowledge of a well-trained network. For example, a series of model compression techniques (Bucilu et al., 2006; Ba & Caruana, 2014; Hinton et al., 2015; Romero et al., 2014) were proposed to fit a lighter network to predict the output of a heavier network. Pre-training (Simonyan & Zisserman, 2014) was adopted to preinitialize certain layers of a deeper network with weights learned from a shallower network. However, network morphism requires the knowledge being fully transferred, and existing work includes Net2Net (Chen et al., 2015) and NetMorph (Wei et al., 2016). Net2Net achieved this goal by padding identity mapping layers into the neural network, while NetMorph decomposed a convolutional layer into two layers by deconvolution. Note that the network morphism operations in (Chen et al., 2015; Wei et al., 2016) are quite primitive and at a micro-scale layer level. In this research, we study the network morphism at a meso-scale module level, and in particular, we investigate its morphing capability.
Modularized Network Architecture. The evolution of convolutional neural networks has been from sequential to modularized. For example, LeNet (LeCun et al., 1998), AlexNet (Krizhevsky et al., 2012), and VGG net (Simonyan & Zisserman, 2014) are sequential networks, and their difference is primarily on the number of layers, which is 5, 8, and up to 19 respectively. However, recently
proposed networks, such as GoogLeNet (Szegedy et al., 2014; 2015) and ResNet (He et al., 2015), follow a modularized architecture design, and have achieved the state-of-the-art performance. This is why we wish to study network morphism at the module level, so that its operations are able to directly apply to these modularized network architectures.
3 NETWORK MORPHISM VIA GRAPH ABSTRACTION
In this section, we present a systematic study on the capability of network morphism learning scheme. We shall verify that a convolutional layer is able to be morphed into any single-source, single-sink DAG subnet, named as a module here. We shall also present the corresponding morphing algorithms.
For simplicity, we first consider convolutional neural networks with only convolutional layers. All other layers, including the non-linearity and batch normalization layers, will be discussed later in this paper.
3.1 BACKGROUND AND BASIC NOTATIONS
For a 2D deep convolutional neural network (DCNN), as shown in Fig. 1a, the convolution is defined by: Bj(cj) = ∑ ci Bi(ci) ∗Gl(cj , ci), (1)
where the blob B∗ is a 3D tensor of shape (C∗, H∗,W∗) and the convolutional filter Gl is a 4D tensor of shape (Cj , Ci,Kl,Kl). In addition, C∗, H∗, and W∗ represent the number of channels, height and width of B∗. Kl is the convolutional kernel size2.
In a network morphism process, the convolutional layer Gl in the parent network is morphed into two convolutional layers Fl and Fl+1 (Fig. 1a), where the filters Fl and Fl+1 are 4D tensors of shapes (Cl, Ci,K1,K1) and (Cj , Cl,K2,K2). This process should follow the morphism equation:
G̃l(cj , ci) = ∑ cl Fl(cl, ci) ∗ Fl+1(cj , cl), (2)
where G̃l is a zero-padded version of Gl whose effective kernel size is K̃l = K1 +K2 − 1 ≥ Kl. (Wei et al., 2016) showed a sufficient condition for exact network morphism:
max(ClCiK 2 1 , CjClK 2 2 ) ≥ CjCi(K1 +K2 − 1)2. (3)
For simplicity, we shall denote equations (1) and (2) as Bj = Gl ~Bi and G̃l = Fl+1 ~ Fl, where ~ is a non-communicative multi-channel convolution operator. We can also rewrite equation (3) as max(|Fl|, |Fl+1|) ≥ |G̃l|, where | ∗ | measures the size of the convolutional filter.
2Generally speaking, Gl is a 4D tensor of shape (Cj , Ci,KHl ,K W l ), where convolutional kernel sizes for blob height and width are not necessary to be the same. However, in order to simplify the notations, we assume that KHl = K W l , but the claims and theorems in this paper apply equally well when they are different.
3.2 ATOMIC NETWORK MORPHISM
We start with the simplest cases. Two atomic morphing types are considered, as shown in Fig. 1b: 1) a convolutional layer is morphed into two convolutional layers (TYPE-I); 2) a convolutional layer is morphed into two-way convolutional layers (TYPE-II). For the TYPE-I atomic morphing operation, equation (2) is satisfied, while For TYPE-II, the filter split is set to satisfy
Gl = F 1 l + F 2 l . (4)
In addition, for TYPE-II, at the source end, the blob is split with multiple copies; while at the sink end, the blobs are joined by addition.
3.3 GRAPH ABSTRACTION
To simplify the representation, we introduce the following graph abstraction for network morphism. For a convolutional neural network, we are able to abstract it as a graph, with the blobs represented by vertices, and convolutional layers by edges. Formally, a DCNN is represented as a DAG M = (V,E), where V = {Bi}Ni=1 are blobs and E = {el = (Bi, Bj)}Ll=1are convolutional layers. Each convolutional layer el connects two blobs Bi and Bj , and is associated with a convolutional filter Fl. Furthermore, in this graph, if outdegree(Bi) > 1, it implicitly means a split of multiple copies; and if indegree(Bi) > 1, it is a joint of addition.
Based on this abstraction, we formally introduce the following definition for modular network morphism: Definition 1. Let M0 = ({s, t}, e0) represent the graph with only a single edge e0 that connects the source vertex s and sink vertex t. M = (V,E) represents any single-source, single-sink DAG with the same source vertex s and the same sink vertex t. We call such an M as a module. If there exists a process that we are able to morph M0 to M , then we say that module M is morphable, and the morphing process is called modular network morphism.
Hence, based on this abstraction, modular network morphism can be represented as a graph transformation problem. As shown in Fig. 2b, module (C) in Fig. 2a can be transformed from module M0 by applying the illustrated network morphism operations.
For each modular network morphing, a modular network morphism equation is associated: Definition 2. Each module essentially corresponds to a function from s to t, which is called a module function. For a modular network morphism process from M0 to M , the equation that guarantees the module function unchanged is called modular network morphism equation.
It is obvious that equations (2) and (4) are the modular network morphism equations for TYPE-I and TYPE-II atomic morphing types. In general, the modular network morphism equation for a module M is able to be written as the sum of all convolutional filter compositions, in which each composition is actually a path from s to t in the module M . Let {(Fp,1, Fp,2, · · · , Fp,ip) : p = 1, · · · , P, and ip is the length of path p} be the set of all such paths represented by the convolutional filters. Then the modular network morphism equation for module M can be written as
Gl = ∑ p Fp,ip ~ Fp,ip−1 ~ · · ·~ Fp,1. (5)
As an example illustrated in Fig. 2a, there are four paths in module (D), and its modular network morphism equation can be written as
Gl = F5 ~ F1 + F6 ~ (F3 ~ F1 + F4 ~ F2) + F7 ~ F2, (6)
where Gl is the convolutional filter associated with e0 in module M0.
3.4 THE COMPATIBILITY OF NETWORK MORPHISM EQUATION
One difficulty in this graph abstraction is in the dimensional compatibility of convolutional filters or blobs. For example, for the TYPE-II atomic morphing in Fig. 1b, we have to satisfy Gl = F 1l +F 2 l . Suppose that Gl and F 2l are of shape (64, 64, 3, 3), while F 1 l is (64, 64, 1, 1), they are actually not addable. Formally, we define the compatibility of modular network morphism equation as follows:
Definition 3. The modular network morphism equation for a module M is compatible if and only if the mathematical operators between the convolutional filters involved in this equation are welldefined.
In order to solve this compatibility problem, we need not to assume that blobs {Bi} and filters {Fl} are finite dimension tensors. Instead they are considered as infinite dimension tensors defined with a finite support3, and we call this as an extended definition. An instant advantage when we adopt this extended definition is that we will no longer need to differentiate Gl and G̃l in equation (2), since G̃l is simply a zero-padded version of Gl.
Lemma 4. The operations + and ~ are well-defined for the modular network morphism equation. Namely, if F 1 and F 2 are infinite dimension 4D tensors with finite support, let G = F 1 + F 2 and H = F 2 ~ F 1, then both G and H are uniquely defined and also have finite support.
Sketch of Proof. It is quite obvious that this lemma holds for the operator +. For the operator ~, if we have this extended definition, the sum in equation (2) will become infinite over the index cl. It is straightforward to show that this infinite sum converges, and also that H is finitely supported with respect to the indices cj and ci. Hence H has finite support.
As a corollary, we have:
Corollary 5. The modular network morphism equation for any module M is always compatible if the filters involved in M are considered as infinite dimension tensors with finite support.
3.5 SIMPLE MORPHABLE MODULES
In this section, we introduce a large family of modules, i.e, simple morphable modules, and then provide their morphing solutions. We first introduce the following definition:
Definition 6. A module M is simple morphable if and only if it is able to be morphed with only combinations of atomic morphing operations.
Several example modules are shown in Fig. 2a. It is obvious that modules (A)-(C) are simple morphable, and the morphing process for module (C) is also illustrated in Fig. 2b.
For a simple morphable module M , we are able to identity a morphing sequence from M0 to M . The algorithm is illustrated in Algorithm 1. The core idea is to use the reverse operations of atomic morphing types to reduce M to M0. Hence, the morphing process is just the reverse of the reduction process. In Algorithm 1, we use a four-element tuple (M, e1, {e2, e3}, type) to represent the process of morphing edge e1 in module M to {e2, e3} using TYPE-<TYPE> atomic operation. Two auxiliary functions CHECKTYPEI and CHECKTYPEII are further introduced. Both
3A support of a function is defined as the set of points where the function value is non-zero, i.e., support(f) = {x|f(x) 6= 0}.
Algorithm 1 Algorithm for Simple Morphable Modules Input: M0; a simple morphable module M Output: The morphing sequence Q that morphs M0 to M using atomic morphing operations Q = ∅ while M 6= M0 do
while CHECKTYPEI(M ) is not FALSE do // Let (Mtemp, e1, {e2, e3}, type) be the return value of CHECKTYPEI(M ) Q.prepend((Mtemp, e1, {e2, e3}, type)) and M ←Mtemp end while while CHECKTYPEII(M ) is not FALSE do
// Let (Mtemp, e1, {e2, e3}, type) be the return value of CHECKTYPEII(M ) Q.prepend((Mtemp, e1, {e2, e3}, type)) and M ←Mtemp
end while end while
Algorithm 2 Algorithm for Irreducible Modules Input: Gl; an irreducible module M Output: Convolutional filters {Fi}ni=1 of M Initialize {Fi}ni=1 with random noise. Calculate the effective kernel size of M , expand Gl to G̃l by padding zeros. repeat
for j = 1 to n do Fix {Fi : i 6= j}, and calculate Fj = deconv(G̃l, {Fi : i 6= j}) Calculate loss l = ‖G̃l − conv({Fi}ni=1)‖2
end for until l = 0 or maxIter is reached
of them return either FALSE if there is no such atomic sub-module in M , or a morphing tuple (M, e1, {e2, e3}, type) if there is. The algorithm of CHECKTYPEI only needs to find a vertex satisfying indegree(Bi) = outdegree(Bi) = 1, while CHECKTYPEII looks for the matrix elements > 1 in the adjacent matrix representation of module M .
Is there a module not simple morphable? The answer is yes, and an example is the module (D) in Fig. 2a. A simple try does not work as shown in Fig. 2b. In fact, we have the following proposition:
Proposition 7. Module (D) in Fig. 2a is not simple morphable.
Sketch of Proof. A simple morphable module M is always able to be reverted back to M0. However, for module (D) in Fig. 2a, both CHECKTYPEI and CHECKTYPEII return FALSE.
3.6 MODULAR NETWORK MORPHISM THEOREM
For a module that is not simple morphable, which is called a complex module, we are able to apply Algorithm 1 to reduce it to an irreducible module M first. For M , we propose Algorithm 2 to solve the modular network morphism equation. The core idea of this algorithm is that, if only one convolutional filter is allowed to change with all others fixed, the modular network morphism equation will reduce to a linear system. The following argument guarantees the correctness of Algorithm 2.
Correctness of Algorithm 2. Let Gl and {Fi}ni=1 be the convolutional filter(s) associated with M0 and M . We further assume that one of {Fi}, e.g., Fj , is larger or equal to G̃l, where G̃l is the zeropadded version of Gl (this assumption is a strong condition in the expanding mode). The module network morphism equation for M can be written as
G̃l = C1 ~ Fj ~ C2 + C3, (7)
where C1, C2, and C3 are composed of other filters {Fi : i 6= j}. It can be checked that equation (7) is a linear system with |G̃l| constraints and |Fj | free variables. Since we have |Fj | ≥ |G̃l|, the system is non-deterministic and hence solvable as random matrices are rarely inconsistent.
For a general module M , whether simple morphable or not, we apply Algorithm 1 to reduce M to an irreducible module M ′, and then apply Algorithm 2 to M ′. Hence we have the following theorem:
Theorem 8. A convolutional layer can be morphed to any module (any single-source, single-sink DAG subnet).
This theorem answers the core question of network morphism, and provides a theoretical upper bound for the capability of this learning scheme.
3.7 NON-LINEARITY AND BATCH NORMALIZATION IN MODULAR NETWORK MORPHISM
Besides the convolutional layers, a neural network module typically also involves non-linearity layers and batch normalization layers, as illustrated in Fig. 3. In this section, we shall describe how do we handle these layers for modular network morphism.
For the non-linear activation layers, we adopt the solution proposed in (Wei et al., 2016). Instead of directly applying the non-linear activations, we are using their parametric forms. Let ϕ be any non-linear activation function, and its parametric form is defined to be
P -ϕ = {ϕa}|a∈[0,1] = {(1− a) · ϕ+ aϕid}|a∈[0,1]. (8)
The shapes of the parametric form of the non-linear activation ϕ is controlled by the parameter a. When a is initialized (a = 1), the parametric form is equivalent to an identity function, and when the value of a has been learned, the parametric form will become a non-linear activation. In Fig. 3b, the non-linear activation for the morphing process is annotated as PReLU to differentiate itself with the
other ReLU activations. In the proposed experiments, for simplicity, all ReLUs are replaced with PReLUs.
The batch normalization layers (Ioffe & Szegedy, 2015) can be represented as
newdata = data−mean√
var + eps · gamma+ beta. (9)
It is obvious that if we set gamma = √ var + eps and beta = mean, then a batch normalization layer is reduced to an identity mapping layer, and hence it can be inserted anywhere in the network. Although it is possible to calculate the values of gamma and beta from the training data, in this research, we adopt another simpler approach by setting gamma = 1 and beta = 0. In fact, the value of gamma can be set to any nonzero number, since the scale is then normalized by the latter batch normalization layer (lower right one in Fig. 3b). Mathematically and strictly speaking, when we set gamma = 0, the network function is actually changed. However, since the morphed filters for the convolutional layers are roughly randomized, even though the mean of data is not strictly zero, it is still approximately zero. Plus with the fact that the data is then normalized by the latter batch normalization layer, such small perturbation for the network function change can be neglected. In the proposed experiments, only statistical variances in performance are observed for the morphed network when we adopt setting gamma to zero. The reason we prefer such an approach to using the training data is that it is easier to implement and also yields slightly better results when we continue to train the morphed network.
4 EXPERIMENTAL RESULTS
In this section, we report the results of the proposed morphing algorithms based on current state-ofthe-art ResNet (He et al., 2015), which is the winner of 2015 ImageNet classification task.
4.1 NETWORK ARCHITECTURES OF MODULAR NETWORK MORPHISM
We first introduce the network architectures used in the proposed experiments. Fig. 3a shows the module template in the design of ResNet (He et al., 2015), which is actually a simple morphable two-way module. The first path consists of two convolutional layers, and the second path is a shortcut connection of identity mapping. The architecture of the ResNet module can be abstracted as the graph in Fig. 4a. For the morphed networks, we first split the identity mapping layer in the ResNet module into two layers with a scaling factor of 0.5. Then each of the scaled identity mapping layers is able to be further morphed into two convolutional layers. Fig. 3b illustrates the case with only one scaled identity mapping layer morphed into two convolutional layers, and its equivalent graph abstraction is shown in Fig. 4b. To differentiate network architectures adopted in this research, the notation morph <k1>c<k2> is introduced, where k1 and k2 are kernel sizes in the morphed network. If both of scaled identity mapping branches are morphed, we append a suffix of ‘ 2branch’. Some examples of morphed modules are illustrated in Fig. 4. We also use the suffix ‘ half’ to indicate that only one half (odd-indexed) of the modules are morphed, and the other half are left as original ResNet modules.
4.2 EXPERIMENTAL RESULTS ON THE CIFAR10 DATASET
CIFAR10 (Krizhevsky & Hinton, 2009) is a benchmark dataset on image classification and neural network investigation. It consists of 32×32 color images in 10 categories, with 50,000 training images and 10,000 testing images. In the training process, we follow the same setup as in (He et al., 2015). We use a decay of 0.0001 and a momentum of 0.9. We adopt the simple data augmentation with a pad of 4 pixels on each side of the original image. A 32×32 view is randomly cropped from the padded image and a random horizontal flip is optionally applied.
Table 1 shows the results of different networks morphed from ResNet (He et al., 2015). Notice that it is very challenging to further improve the performance, for ResNet has already boosted the number to a very high level. E.g., ResNet (He et al., 2015) made only 0.36% performance improvement by extending the model from 56 to 110 layers (Table 1). From Table 1 we can see that, with only 1.2x or less computational cost, the morphed networks achieved 2.15%, 1.60%, 1.11% performance improvements over the original ResNet-20, ResNet-56, and ResNet-110 respectively. Notice that the relative performance improvement can be up to 25%. Table 1 also compares the number of parameters of the original network architectures and the ones after morphing. As can be seen, the morphed ones only have a little more parameters than the original ones, typically less than 1.2x.
Except for large error rate reduction achieved by the morphed network, one exciting indication from Table 1 is that the morphed 20-layered network morph20 3c3 is able to achieve slightly lower error rate than the 110-layered ResNet (6.60% vs 6.61%), and its computational cost is actually less than 1/5 of the latter one. Similar results have also been observed from the morphed 56-layered network. It is able to achieve a 5.37% error rate, which is even lower than those of ResNet-110 (6.61%) and ResNet-164 (5.46%) (He et al., 2016). These results are also illustrated in Fig. 5(a).
Several different architectures of the morphed networks were also explored, as illustrated in Fig. 4 and Table 1. First, when the kernel sizes were expanded from 1 × 1 to 3 × 3, the morphed networks (morph20 3c1 and morph20 3c3) achieved better performances. Similar results were reported in (Simonyan & Zisserman, 2014) (Table 1 for models C and D). However, because the morphed networks almost double the computational cost, we did not adopt this approach. Second, we also tried to morph the other scaled identity mapping layer into two convolutional layers (morph20 1c1 2branch), the error rate was further lowered for the 20-layered network. However, for the 56-layered and 110-layered networks, this strategy did not yield better results.
We also found that the morphed network learned with multiple phases could achieve a lower error rate than that learned with single phase. For example, the networks morph20 3c1 and morph20 3c3 learned with intermediate phases achieved better results in Table 1. This is quite reasonable as it divides the optimization problem into sequential phases, and thus is possible to avoid being trapped into a local minimum to some extent. Inspired by this observation, we then used a 1c1 half network as an intermediate phase for the morph56 1c1 and morph110 1c1 networks, and better results have been achieved.
We compared the proposed learning scheme against learning from scratch for the networks with the same architectures. These results are illustrated in Table 2. As can be seen, networks learned by morphing is able to achieve up to 2.66% absolute performance improvement and 32.6% relative performance improvement comparing against learning from scratch for the morph110 1c1 network architecture. These results are quite reasonable as when networks are learned by the proposed morphing scheme, they have already been regularized and shall have lower probability to be trapped into a bad-performing local minimum in the continual training process than the learning from scratch scheme. One may also notice that, morph110 1c1 actually performed worse than resnet110 when learned from scratch. This is because the network architecture morph 1c1 is proposed for morphing, and the identity shortcut connection is scaled with a factor of 0.5. It was also reported that residual networks with a constant scaling factor of 0.5 actually led to a worse performance in (He et al., 2016) (Table 1), while this performance degradation problem could be avoided by the proposed morphing scheme.
Finally, it is worth noting that another advantage of the proposed learning scheme against the learning from scratch scheme is on model exploration. One can quickly check whether a morphed architecture deserves further exploration by continuing to train the morphed network in a finer learning rate (e.g. 1e-5), to see if the performance is improved. Hence, one does not have to wait for days or even months of training time to tell whether the new network architecture is able to achieve a
better performance. This could save human time for deciding which network architecture is worth for exploring.
4.3 EXPERIMENTAL RESULTS ON THE CIFAR100 DATASET
CIFAR100 (Krizhevsky & Hinton, 2009) is another benchmark dataset for tiny images that consists of 100 categories. There are 500 training images and 100 testing images per category. The proposed experiments on CIFAR100 follows the same setup as in the experiments on CIFAR10. The experimental results are illustrated in Table 3 and Fig. 5(b). As shown, the performance improvement is also significant: with only around 1.1x computational cost, the absolute performance improvement can be up to 2% and the relative performance improvement can be up to 8%. For the morphed 56-layered network, it also achieves better performance than the 110-layered ResNet (27.52% vs 28.46%), and with only around one half of the computation. Table 4 also compares the proposed learning scheme against learning from scratch. More than 5% absolute performance improvement and around 16% relative performance improvement were achieved.
4.4 EXPERIMENTAL RESULTS ON THE IMAGENET DATASET
We also evaluate the proposed scheme on the ImageNet dataset (Russakovsky et al., 2014). This dataset consists of 1,000 object categories, with 1.28 million training images and 50K validation images. For the training process, we use a decay of 0.0001 and a momentum of 0.9. The image is resized to guarantee its shorter edge is randomly sampled from [256,480] for scale augmentation. A 224× 224 patch or its horizontal flip is randomly cropped from the re-
sized image, with the image data per-channel normalized. We train the networks using SGD with a batch size of 256. The learning rate starts from 0.1 and is decreased with a factor of 0.1 for every 30 epochs. The networks are trained for a total of 70 epochs.
The comparison results of the morphed and original ResNets for both 18-layer and 34-layer networks are illustrated in Table 5 and Fig. 6. As shown in Table 5, morph18 1c1 and morph34 1c1 are able to achieve lower error rates than ResNet-18 and ResNet-34 respectively, and the absolute performance improvements can be up to 1.2%. We also draw the evaluation error curves in Fig. 6, which shows that the morphed networks morph18 1c1 and morph34 1c1 are much more effective than the original ResNet-18 and ResNet-34 respectively.
5 CONCLUSIONS
This paper presented a systematic study on the problem of network morphism at a higher level, and tried to answer the central question of such learning scheme, i.e., whether and how a convolutional layer can be morphed into an arbitrary module. To facilitate the study, we abstracted a modular network as a graph, and formulated the process of network morphism as a graph transformation process. Based on this formulation, both simple morphable modules and complex modules have been defined and corresponding morphing algorithms have been proposed. We have shown that a convolutional layer can be morphed into any module of a network. We have also carried out experiments to illustrate how to achieve a better performing model based on the state-of-the-art ResNet with minimal extra computational cost on benchmark datasets. The experimental results have demonstrated the effectiveness of the proposed morphing approach.

This paper studies knowledge transfer problem from small capacity network to bigger one. This is a follow-up work of Net2Net (ICLR 2015) and NetMorph(ICML 2016).
Comments
- 1) This paper studies macroscopic problem, with the morphing process composed by multiple atomic operations. While the atomic operations are proposed in Net2Net and NetMorph, there has not been study of the general modularized process principally. Thus this paper asks a novel question.
- 2) The solution by composing multiple atomic transformations seems to be quite reasonable.
- 3) In the “related work” section, it is better to change “network morphism” to “knowledge transfer” or in the subsection title, most of these works are known as knowledge transfer and it helps to connect to the existing works.
- 4) The author shows experiments on variants of ResNet. While the experiment shows that initializing from ResNet gives better error rate than the ones trained from scratch, it is unclear what the source This paper studies knowledge transfer problem from small capacity network to bigger one. This is a follow-up work of Net2Net (ICLR 2015) and NetMorph(ICML 2016).
Comments
- 1) This paper studies macroscopic problem, with the morphing process composed by multiple atomic operations. While the atomic operations are proposed in Net2Net and NetMorph, there has not been study of the general modularized process principally. Thus this paper asks a novel question.
- 2) The solution by composing multiple atomic transformations seems to be quite reasonable.
- 3) In the “related work” section, it is better to change “network morphism” to “knowledge transfer” or in the subsection title, most of these works are known as knowledge transfer and it helps to connect to the existing works.
- 4) The author shows experiments on variants of ResNet. While the experiment shows that initializing from ResNet gives better error rate than the ones trained from scratch, it is unclear what the source is.
- 5) One major advantage of this type of knowledge transfer (Net2Net, NetMorph) is to speedup training and model exploration. There seems to be no experiments demonstrate such advantage (possibly due to the lose initialization of BatchNorm). This is the major drawback of this paper.
-6)  The method proposed by the author can in principle do quite complicated transformation, e.g. transform  an entire resnet from a single conv layer, the experiment only consists of simple module transformations, which in some way can be covered by atomic operations. It would be more interesting to see what the results of more complicated transformations are (even if they are not as effective).
In summary, this paper studies a novel problem of knowledge transfer in a macroscopic level. The method could be of interest to the ICLR community. The experiments should be improved (comment 5) to make the results more convincing and practically useful and I strongly encourage the authors to do so.


Training a classifier with only a few examples remains a significant barrier when using neural networks with a large number of parameters. Though various specialized network architectures have been proposed for these k-shot learning tasks to avoid overfitting, a question remains: is there a generalizable framework for the k-shot learning problem that can leverage existing deep models as well as avoid model overfitting? In this paper, we proposed a generalizable k-shot learning framework that can be used on any pre-trained network, by grouping network parameters to produce a low-dimensional representation of the parameter space. The grouping of the parameters is based on an orthogonal decomposition of the parameter space. To avoid overfitting, groups of parameters will be updated together during the k-shot training process. Furthermore, this framework can be integrated with any existing popular deep neural networks such as VGG, GoogleNet, ResNet, without any changes in the original network structure or any sacrifices in performance. We evaluate our framework on a wide range of k-shot learning tasks and show state-of-the-art performance.
1 INTRODUCTION
As many deep learning network architectures have been proposed, several key models have emerged as default models for many image classification tasks. In particular, deep neural network architectures such as the VGG network by Karen & Zisserman (2014), Inception by Christian et al. (2015) and ResNets by Kaiming et al. (2015), have proven their superb performance on image classification when using datasets such as ImageNet (Alex et al. (2012)) and MS COCO (Lin et al. (2014)). With enough data and fine tuning, these ‘go to’ models have been shown to be successful for many visual classification tasks.
However, there is a problem when one does not have access to a large labeled training dataset to fine-tune these models. This task of training a classifier using only a small k number of examples is often referred to as k-shot learning and has problems when dealing with high capacity models such as deep convolutional neural networks. The problem of training with only a small number of training examples is that it often leads to overfitting, where the model essentially memories those data points without gaining the ability to generalize to new instances.
Current state-of-the-art methods (Oriol et al. (2016) Adam et al. (2016)) apply deep learning techniques by using specialized network architectures for k-shot learning to avoid overfitting. While this is a reasonable strategy for proposing diverse kinds of new frameworks for k-shot learning tasks, it is hard for different k-shot learning methods to borrow structures from each other since they are all highly customized networks.
We propose a method called the Orthogonal Method of Grouping (OMG) to facilitate a better k-shot learning process. OMG ensures that similar (near duplicate) features in the classifier will be grouped and modified together during the training process. This process of grouping features naturally induces dimension reduction of the parameter space and imposes a form of subspace regularization during training. We implement OMG by adding a new loss layer that essentially clusters (groups) parameters by slighting perturbing them according to an orthogonality constraint. This para-loss layer only augments the network and does not require any changes to the original architecture. Once the
feature has been grouped, the network can be used to learn from only a few examples and parameter updates are propagated to parameter groups instead of individual parameter updates.
Our contribution is threefold: (1) we proposed a general k-shot learning approach which does not rely on any task-specific prior knowledge; (2) our approach can be added to any network without changing the original network structure; and (3) the proposed method provides an effective technique for decomposing the parameter space for high capacity classifiers.
2 RELATED WORKS
One-shot learning is an interesting topic first presented in Li et al. (2006). The key idea of oneshot learning is to make a prediction on a test instance by only observing a few examples of oneshot classes before. Li et al. (2006) solved this problem by adopting Variational Bayesian approach where object categories are represented by probabilistic models. More recently, researchers revisited one-shot learning with highly customized models: M. et al. (2011) address one-shot learning for character recognition with a method called Hierarchical Bayesian Program Learning (HBPL). It modeled the process of drawing characters generatively to decompose the image into small pieces. The goal of HBPL is to determine a structural explanation for observed pixels. However, inference under HBPL is difficult since the joint parameter space is very large, which leads to an intractable integration problem. Gregory (2015) also presented a strategy for performing one-shot classification by learning deep convolutional siamese neural network for verification. Oriol et al. (2016) proposed a Matching Nets utilizing external memory with attention kernel. Adam et al. (2016) proposed the memory-augmented neural networks with an external content based memory. These works achieve state-of-the-art performance on specific datasets. However, one important issue would be, works mentioned above proposed highly customized networks, whose structures is hard to be borrowed from other. Conversely, OMG is a general one-shot learning model can fit into any existing networks. So its structure could borrow by any other works.
Domain Adaptation is another related topic to our work. It aims at learning from a source data distribution a well-performing model on a different (but related) target data distribution. Many successful works as Boqing et al. (2012) and Basura et al. (2013) seek an embedding of transformation from the source to target point that minimizes domain shift. Daume III (Daum III (2009)) is another simple feature replication method that augments feature vectors with a source component, a target component, and a shared component. Then an SVM is trained on the augmented source and target data. These methods are proven to be effective for many tasks, but none of these methods above could feed into an end to end learning framework. One end to end learning framework would be the supervised adaptation method proposed by Judy et al. (2013a). It trains different deep networks for source and target domain and concatenating the high-level features as final embedding. However, the limitation is - the high-level features trained by one domain can not borrow knowledge from the other domain. In contrast, our method is an end-to-end method learning on target data without training additional networks. So the OMG can contain knowledge from both source and target domain, which is to say, the source and target domain could borrow knowledge from each other.
Our OMG model could decompose existing feature to a compact feature space, which is closely related to dimension reduction. Many cookbook works such as Aapo (1999) Jolliffe. (1986) Najim et al. (2011) have been studied extensively in the past. However, these works can’t integrate into any current end-to-end deep learning frameworks. As far as we know, there is limited work having visited the topic of end-to-end dimension reduction. E. & Salakhutdinov (2006) used auto-encoder as a dimension reduction method. Stacked restricted Boltzmann machine (RBM) is proposed to embed images to lower dimension. However, this work is also hard to be integrated into existing network structures or utilize pre-trained models. The reason is that it has a different architecture than CNN networks, and it requires to train from scratch. Our OMG is an end-to-end approach that’s able to feed into any architectures smoothly for both training and finetuning. Furthermore, OMG could reduce the output to an arbitrary dimension without changing the architecture and sacrifice the performance.
3 APPROACH
We observe the fact that the parameters of each layer of the deep network are always correlated. This can be best illustrated by Fig. 1. The correlated parameters will result in correlated outputs with a lower capacity. When learning new classifier on top of these correlated outputs, it is easier to get an instance specific classifier if only small amount of data is seen. In most of the cases, an instance specific classifier is not what we always want. If the correlation of output is removed, which output is becoming orthogonal, then a better classifier will be fetched given a few data. So we proposed an Orthogonal Method of Grouping (OMG) to remove the correlation of outputs by decomposing the correlated the parameters to orthogonal parameters.
3.1 ORTHOGONAL METHOD OF GROUPING
OMG is a two-step method which can be best illustrated by looking at Figure 2. In Figure 2 (a), each purple vector represent a parameter.
In the first step of OMG (Figure 2 (b)), it finds correlated parameters and groups them in the same subspace. Since correlated parameters will result in correlated outputs, we can find the relation between parameters by analysis the output (e.g., the relation between convolutional kernels could be found by analysis the activation of the convolutional layers). In the second step of OMG (Figure 2 (c)), each parameter is slightly perturbed such that the orthogonality between each grouped subspace is maximized. The groups are represented as an Orthogonal Group Mapping that each parameter is mapped to the corresponding group. The mapping is learned by optimizing the orthogonal constraint on both θw and θmap, where θw is the parameters of the neural networks, and θmap is the parameters of the Orthogonal Group Mapping.
The Orthogonal Group Mapping is firstly introduced in Sec 3.1.1, followed with loss function and orthogonal constraint in Sec 3.1.2. Then the optimizing algorithm is introduced in Sec 3.1.3. Finally, the k-shot learning method learned on orthogonal grouped parameters is introduced in 3.2
3.1.1 ORTHOGONAL GROUP MAPPING
Orthogonal Group Mapping (OGM) maps neural units to the corresponding groups by a mapping. Let neural units in a layer of the network be denoted as a set n = {n1, n2, ..., nL}, where L is the number of neural unit in that layer. (For example, a filter of a fully convolutional layer can represent as a unit.) The Orthogonal Group Mapping could represent as orthogonal group sets g, where gk is the kth orthogonal group in g. Each orthogonal group gi contains the corresponding units gi = {nj , ..., nl}. Since the OGM is a map that, one unit is only mapped to one single group.
3.1.2 PARA-LOSS
We cast constraints to force orthogonal groups orthogonal to each other. The constraint is achieve by a loss function with two terms: intra-class loss Lintra and inter-class class Linter. Lintra minimize the divergence in each group, and Linter force the basis in each orthogonal group orthogonal to each other.
When a mini-batch of input data of sizeB is propagated through the network, the output (activation) of units n over the mini-batch can be denoted as a matrix A ∈ RB×L, where each element Al ∈ R1×L denotes the output of the l-th unit over the entire mini-batch. Intuitively, if the outputs Ai and Aj of two units ni, nj are similar, we would like to let the two units belong to the same orthogonal group gk. Conversely if the outputs Ai and Aj of two units ni, nj are different, then they should belong to different orthogonal groups.
We define the intra-group loss by the sum of squared distances between each orthogonal groups:
Lintra = ∑ k ∑ i,j∈gk ||Ai −Aj ||2
The time complexity is Θ(Lintra) = K × kmax × kmax, where kmax = maxi ki. This could be efficiently computed when kmax is small, but we still want to reduce Θ(Lintra) since the computational cost of this loss can be significant when there are many units in a single layer. We can use a lower bound to approximate the distance:
L̃intra = ∑ k ∑ i∈gk(Ai −Aanchor) 2
where anchor is an index randomly selected from the k-th orthogonal group. This approximation reduces the time complexity to Θ(Lintra) = K × kmax ≈ L, which is linear. In addition to quantifying the intra-group similarity (tightness of each cluster), we also want to measure the separation between each group. We can define an inter group loss in terms of an orthogonality measure:
Linter = ∑ i,j ‖M>i Mj‖2F
Where matrix Mi ∈ RB×|gi| represents the output of all the units in orthogonal group gi over the mini-batch, where |gi| is the number of units in the orthogonal group gi. That is to say, Mi = [Aj , ..., Ak]
T , where nj , nk ∈ gi. The ‖ · ‖2F is the squared Frobenius norm. This term is minimized when feature vectors are exactly orthogonal.
The entire para-loss function is denoted as: L = α ∑ k ∑ i∈gk(Ai −Aanchor) 2 + β ∑ i,j ‖M>i Mj‖2F
It is easy to see theL does not contain any term of ground truth label. So the OMG is an unsupervised method without requiring ground truth. However, this method could work smoothly with other losses. For example, OMG could train with loss of softmax Lsoftmax simultaneously for a supervise learning task.
3.1.3 OPTIMIZATION
We proposed optimizing method for OMG to optimize the constraint on both parameters of the neural networks θw, and parameters of the Orthogonal Group Mapping θmap. We use a two-step approach to optimize argminθmap,θwL by optimizing argminθwL and argminθmapL iteratively, where θw, is the parameter of original network, and θmap is the parameter of Orthogonal Group Mapping. For the first step, we optimize the weights of the net θw with SGD. For the second step, we optimize the weights of mapping θmap with Algorithm 1.
First step: We use the standard SGD to optimize argminθwL: θw := θw − η∇(θw) = θw − η ∑ i Li(θw)
Where η is the step size, and Li(θw) is the value of the loss function at ith iteration.
Second step: We propose Algorithm 1 to optimize argminθmapL:
Algorithm 1 Optimization Algorithm Initialization Random Initialize Mapping θmap Given a batch of training set for each iteration do
for each group gi, i ∈ [1,K] do Find the max violated unit nl in gi that l ∈ gi by: l = argmaxl ∑ m∈gi(Al −Am) 2
Reassign the unit nl from gi to new group gk where: k = argmink ∑ m∈gk(Al −Am) 2
end end
It is easy to see that, optimizing argminθmapL will not change any parameter of the original deep network θw. That is to say, given any pre-train network, we could switch the network to an orthogonal grouping network without changing its original parameters.
3.2 DIMENSION REDUCTION AND k-SHOT LEARNING
Given an Orthogonal Method of Grouping with corresponding θw and θmap, we could assign L outputs to K groups. The outputs in each group is correlated. we assign an additional weights wadd ∈ RK and bias badd ∈ RK corresponding to each group. We omit the bias for simplicity. For each group gi it shares the same weight waddi . While doing k-shot learning on a few samples, the original weights θw are fixed, and only the wadd is update. In another view, we could regard this as an end-to-end dimension reduction method. It could reduce the original output dimension L to arbitrary dimension K. Each dimension in the K is largely orthogonal to each other.
4 DATASET
The Orthogonal Method of Grouping is evaluated on three standard datasets: ImageNet, MNIST and Office Dataset. ImageNet (Alex et al. (2012)) is the largest publicly available dataset with image category labels. The MNIST (LeCun et al. (1998)) dataset of handwritten digits contains a training set of 60K examples and a test set of 10K examples. The Office (Saenko et al. (2010)) dataset is a collection of images from three distinct domains: Amazon, DSLR, and Webcam. The dataset contains objects of 31 categories commonly spotted in office, such as keyboards, file cabinets, and laptops. Among the 31 categories, there are 16 overlaps with the categories present in the 1000- category ImageNet classification task. This dataset is first used by Judy et al. (2013c) for k-shot adaptation task and we follow the same data split of their work.
5 EVALUATION
We evaluate the OMG from three different tasks: training from scratch, finetuning and k-shot learning. We report the performance for training from scratch on MNIST to show that OMG could help to improve the performance by cast orthogonal constraint. Then we smoothly integrated OMG into standard pre-trained networks to show OMG can successfully enhance the orthogonality among groups of parameters on arbitrary neural networks, and results in more discriminative parameters. For k-shot learning tasks, we report our k-shot learning results on MNIST and Office Datasets (Saenko et al. (2010)). Experiments show our learned compact orthogonal parameters could facilitate learning classifier on limited data.
In the following sections, Experiments for training from scratch are reported in 5.1, Experiments for finetuning are reported in 5.2 and the k-shot learning experiments are reported in Sec. 5.3.
5.1 TRAINING FROM SCRATCH
We show that our OMG facilitates the performance of neural network during the training. We trained a standard Convolutional Neural Networks (Y. et al. (2003), which reported achieving 1.19% error rates) on MNIST dataset as a baseline. For OMG model, we report the difference of accuracies with different α, β, and group size. The difference of accuracy here denotes the difference between baseline’s accuracy and the proposed model’s accuracy. For example, if the proposed model has 98% of accuracy and the baseline is 97%, then the difference of accuracy is 1. OMG is used to train every convolutional and fully connected layer in the baseline convolutional network rather than one specific layer.
We train OMG from scratch with the different set of α and β. We set α or β to 0, 1e−6, 1e−5, 1e−4, 1e−3, 1e−2, 1e−1 separately, and keep the other hyperparameter to 0. The group size is set to half of the neural unit’s size when evaluating the effectiveness of α and β. We report the difference of accuracy in Table. 1 and Fig. 4. It is easy to see that when α and β ∈ (0, 1e−3), the OMG can boost the performance of the convolutional neural network. We find when the value of hyperparameters is around 1e−3, the generated gradient from L is about 1 - 5% of the gradient from Lsigmoid, which is a reasonable ratio (hyperparameter of L2 normalization is around 1 - 5%). In Fig. 4, it
shows when the hyperparameters are extremely small, the OMG will not change the performance of the network. When the hyperparameters are too large, the constraint will be too strong to learn the parameters properly. When the hyperparameters are in ideal range, then the OMG is casting the orthogonal constraint on the network and force the network to learn more discriminative parameters. This shows the OMG is working smoothly with normal neural networks and able boosting its performance. For the following experiments, we set the α to 5e−5 and β to 1e−4 if we do not specific mentioned. Theoretically, the Lintra would force the filters in each group to be similar, so intuitively, it would jeopardize the performance of the original network. However, as we find in Fig. 4, Lintra can actually boost the performance. This is because practically the Lintra is being used as a regularization term.
We report the effect of group size as shown in Fig. 5. In Fig. 5 when the group size is set as the same as the neural unit size, the OMG is not really grouped. But the network still has a better performance since the OMG enforces the network to learn more discriminative parameters. The best performance is achieved when the group size is around half of the neural unit size. If the group size is too small, it will force all the neural units to learn the same parameters, which would jeopardize the performance of the network. When the group is 1/64 of the neural unit size, it achieves the worst performance. For the following experiments, if we do not mention specifically, we set the group size as half of the neural unit size.
Although the OMG can work smoothly with different types of layers as convolutional layer, fully connected layer, and etc, we only visualize the orthogonal groups of first convolutional layer in Fig. 6 since it is the easiest to observe. It is hard to visualize the rest of convolutional layers since the filter size is smaller. And it is also hard to visualize the fully connect layer since we can not observe
clear pattern directly. For a better visualization, we choose the convolutional neural network with 11 × 11 kernels. Filters in the same blue bounding boxes belong to the same group. We can see filters in each different groups are highly correlated to each other, and the filters in the different group are visually very different.
5.2 FINETUNING
In this section, we prove that OMG forces deep neural network learn more discriminative features, and the OMG works smoothly with pre-train neural networks. We finetune existing pre-trained networks on ImageNet with the help of OMG. The original network is fully trained on ImageNet dataset, so directly finetune on the same dataset should not change any thing significantly. Even finetune on different datasets, empirically the parameters in the very first layers are not going to change. However, we observe the significant changing after finetuning on the same dataset for only 5 epoches.
We choose to visualize VGG-F (Chatfield et al. (2014)) net since it has the largest kernel size among the VGG zoo (11 × 11 rather than 7 × 7). We visualize the grouped filters in the first convolution layer in Fig. 7. It is bacause the same reason, we only visualize the first convoutional layer The kernels in the same vertical line belong to the same group. We have 20 groups and each group has the number of kernels from 1 to 6. It is easy to see that the filters within each group share similar appearance (highly correlated), and the filters from different groups have divergent appearances. For example, the second and the 8th − 13th, they are all white and black edges with the same direction. the kernels in the 6th group are all like gaussian kernels with a little jitter in location.
In order to prove that the OMG can help to learn the more discriminative parameters, we visualize the filters after finetuning in Fig. 8. The filters on the left have strong pattern before finetune, and they do not change much after finetuning. For the filters on the right, the filters do not have the strong patterns, but after finetuning with OMG, the pattern of the filters become more distinct with strongly colorful textures. Our interpretation is, our OMG assigns additional orthogonal constraint to all the filters, force them to learn more discriminative parameters. The filters with strong patterns are not changed too much because they are originally orthogonal to other filters. But for the filters without strong patterns, the orthogonal constraint are highly effective. As a result, the OMG helps the VGG network pretrained on ImageNet to learn the new discriminative paramters during finetuning on the same dataset.
5.3 K-SHOT LEARNING
The performance for k-shot learning is evaluated on two different task: MNIST k-shot learning and Office k-shot learning tasks.
5.3.1 K-SHOT ON MNIST
We perform k-shot learning on MNIST dataset. The performances are evaluated on a 10-way classification where each class is provided with 1, 5 training examples. For the MNIST one shot learning task, we split the data to pre-knowledge set, and one shot learning set with a ratio of 1:9. The models we compared with are k-Nearest Neighbors(K-NN), Support Vector Machines(SVM), Traditional Convolution Neural Networks, and Deep Boltzmann Machines (DBM), and compositional patch model (CPM). The CPM (Alex & Yuille (2015)) is a model designed for learning a compact dictionary of image patches representing meaningful components of an object. The performances were evaluated on a 10-way classification where each class is provided with 1 and 5 training examples to show the growth in accuracy. For a given run, each model is given a set of hand-written digits picked at random from each class from the one-shot learning set. For CNN, we used a structure with four convolutional layers and two pooling layers. For DBM, it contains two hidden layers with 1000 units each. For the OMG model, it learns the grouping information on pre-knowledge set without using ground truth. Then it is trained on the samples from one shot learning set, where the group sizes are set by grid search. The performance is reported in Table. 2. The performance of OMG is better than the baselines, especially better than the previous CNN method. It is because the huge parameter space of original CNN is hard to optimized by only a few samples. But with the help of OMG, the parameter space is significant reduced. The OMG can learn the group of filters representing the shared common parts as each part holds immense amounts of information on how a visual concept is constructed. And using these patches as features to learn a better classifier with limited data.
5.3.2 K-SHOT ON OFFICE DATASET
In this section, we conduct the experiment on k-Shot learning task on Amazon Office Dataset (Saenko et al. (2010)). We followed the work described in Judy et al. (2013c), conduct k-shot learning on the 16 categories of Office dataset (approximately 1,200 examples per category or 20K images total). We evaluated our method across 20 random train/ test splits, and each test split has 160 examples. Then the averages error are reported. For each random train/ test split we choose one example for training and 10 other examples for testing. Following the previous work, we use pre-trained DECAF from ImageNet. The model is additionally trained with Orthogonal Method of Grouping on the last 3 convolutional and fully connect layers, where the group sizes are set by grid search. The group size of each layer is always around half of the neural unit size of the layer. Then we perform k-shot learning on the Office dataset with the reduced dimension. The accuracy is reported in Table. 3. The models we compared with are SVM, PMG, Daume III and Late fusion. Late fusion (Judy et al. (2013b)) is a simple approach to independently train a source and target classifier and combine the scores of the two to create a final scoring function.
We achieve the best performance comparing to previous state-of-the-art method Judy et al. (2013c). With the help of OMG, the dimension reduced network achieve a better performance when training on limited data. The main reason is because, the original feature space on DeCaf is large and redundant. By grouping the feature space, the dimension is largely reduced and the network is still representative enough for the Office Dataset. Then a better performance is achieved on Amazon one shot learning task with OMG.
6 CONCLUSION
We proposed a generalizable k-shot learning framework that can be easily integrated into any existing deep net architectures. By grouping parameters together and forcing orthogonality among groups, the method is able to reduce parameter space dimensionality for avoiding overfitting. Experiments on k-shot learning tasks have proven that OMG is able to have a good performance on k-shot class and easily to be adopted to the current existing deep net framework like VGG, ResNet and so on.

This paper proposes a regularization technique for k-shot learning based on orthogonal grouping of units in a neural network. The units within a group are forced to be maximally similar, at the same time the units from different groups are encouraged to be orthogonal. While I like the motivation of the approach, the empirical analysis provided in the paper doesn’t look particularly convincing.
My main concerns are the following:
1. The method is sensitive to the values of alpha and beta and a poor choice of those hyperparameters can lead to a quite drastic drop in performance comparing the minor gains one gets when alpha and beta are set properly.
2. It seems strange that the best performance is obtained when the group's size ratio is 0.5. From the figures in the paper, it follows that usually, one has more “orthogonal” groups in a filter bank. I have an impression that the empirical evidence doesn’t align well with the motivation of the proposed approach.
3. The paper contains a significant amount of typos and incorrectly formatted references. There are also several places in the manuscript that I found hard to understand due to unusual phrasing.
I would like to thank the authors for answering/addressing my pre-review questions. I would be grateful if the authors could provide more clarifications of the following:
1. Question 2: I’m not sure if modifying \theta_{map} alone would result in any learning at all. Do I understand correctly that \theta_{map} is only used to define groups? If so, then I don’t see how the proposed method can be used in the purely unsupervised regime.
2. Question 3: I was not referring to the fixed clustering based on the filter of the pre-trained network. One can perform that clustering at every step of the k-shot learning process. I’m not sure I understand why the authors visualize grouping of _filters_ while in the actual algorithm they group _activations_.
Overall, the paper is quite interesting but needs a stronger empirical justification of the approach as well as a better presentation of the material.