Multi-Task Zipping via Layer-wise Neuron Sharing
Future mobile devices are anticipated to perceive, understand and react to the world on their own by running multiple correlated deep neural networks on-device. Yet the complexity of these neural networks needs to be trimmed down both withinmodel and cross-model to fit in mobile storage and memory. Previous studies squeeze the redundancy within a single model. In this work, we aim to reduce the redundancy across multiple models. We propose Multi-Task Zipping (MTZ), a framework to automatically merge correlated, pre-trained deep neural networks for cross-model compression. Central in MTZ is a layer-wise neuron sharing and incoming weight updating scheme that induces a minimal change in the error function. MTZ inherits information from each model and demands light retraining to re-boost the accuracy of individual tasks. Evaluations show that MTZ is able to fully merge the hidden layers of two VGG-16 networks with a 3.18% increase in the test error averaged on ImageNet and CelebA, or share 39.61% parameters between the two networks with < 0.5% increase in the test errors for both tasks. The number of iterations to retrain the combined network is at least 17.8× lower than that of training a single VGG-16 network. Moreover, experiments show that MTZ is also able to effectively merge multiple residual networks.
The paper considers the following problem: given a collection of pretrained deep models for different tasks, how to obtain a shared/joint compressed model? Authors work under the assumption that all models have same architecture and propose an approach to identify hidden units across corresponding layers in these models that can be merged together. The criterion for merging looks at the errors introduced in the pre-activations of the units when their corresponding incoming weights are merged -- hidden unit pairs (for the case of two pretrained networks) which result in lowest error are merged. Experiments with LeNets on MNIST and VGG-16 on Imagenet and CelebA are reported to show the effectiveness of the method.   Strengths:  The problem setting considered in the paper seems novel as well as reasonable from a practical perspective.   Weaknesses: The empirical evaluation of the proposed method is not comprehensive enough in my view -  (i) some natural baselines are not considered, eg, the paper starts with trained theta_1... theta_n and merges them in a shared model (theta) - a natural alternative is start with a shared model (either fully shared or usual branching architectures) and train it jointly on all tasks -- how does it perform in comparison to the proposed method on the error vs model-size plane?  (ii) Experiments in Sec 4.1 (two LeNet models, both trained on MNIST) do not convey much information in my view. I am missing the point in merging two models trained on the same task (even with different seeds). (iii) Experiments in sec 4.2 are done the VGG 16 which is a quite heavy model to begin with. It's hard to judge the significance of the results due to lack of baselines. Authors should show model size and accuracy numbers for some recently proposed multitask deep models (for the same set of tasks - imagenet and celebA) to make it easier for the reader.  (iv) All experiments and even the formulation in 3.3 is for two models. How does the method extend to more than two? (v) Algorithm 1 refers to d(w^A, w^B) and f(w^A, w^B). Where are they defined in Sec 3.3? I couldn't find a definition for these.  (vi) The paper claims that retraining needed after the merging is quite fast. What is the time needed for merging? From Algorithm 1, it looks like the method needs L forward passes for all training samples to merge L layers (as after merging a layer 'i', the activations for layer 'i+1' need to be recomputed)?  =================================== I looked at the author response and I still feel many of my concerns are not addressed satisfactorily, particularly the following points that I raised in my initial review.  (i) Natural baseline: I am not fully convinced about the reasons the authors cite for not including this baseline in the first draft. If the results of the baseline were worse than MTZ, it would've been still useful to include the results in the paper and discuss/explain why it performs worse than MTZ in the case of fully shared model. I think this is an interesting observation on its own which deserves discussion, since one would expect joint training of fully shared model to perform better than the two step optimization process of first training individual models and then merging them.   (ii) Same task with different random seeds: I am willing to buy the rationale given in the author response for this. However I would've still liked to see experiments for this setting using more recent architectures that authors use for experiments in Sec 4.2 with same datasets (ideally, ResNets or something more recent..). It is not clear why the paper uses LeNets + mnist.   (iii) More recent architectures and other competitive baselines: Author response has some results on MTZ applied on ResNets. Again, the fully shared baseline is missing from the table. My other point about comparing with recent multitask approaches (which also result in a compressed model) still remains unaddressed. It is difficult to assess the effectiveness of the proposed method (in terms of the trade-off it provides b/w final model size and accuracy) in the absence of any comparison. The absence of any baselines is perhaps the weakest point of the current draft in my view.   (iv) (v) I am fine with the explanation provided in the author response for these.   (vi) Number of fwd passes: If I understand correctly, to apply the approach with a single forward pass, one needs to store *all* the activations of previous layer in the memory so that these can be fwd-propagated after zipping the current layer. I couldn't find details in the paper on how many input examples are used for the zipping process. Author response just mentions "Moreover, only a small part of the whole training dataset is needed to generate an accurate enough Hessian matrix as mentioned in [8].", without giving details on number of examples used in the experiments. Algorithm 1 "input" on the other hand mentions "training datum of task A and B" which gives a feeling that full data is used during MTZ.