We present Group-size Series (GroSS) decomposition, a mathematical formulation of tensor factorisation into a series of approximations of increasing rank terms. GroSS allows for dynamic and differentiable selection of factorisation rank, which is analogous to a grouped convolution. Therefore, to the best of our knowledge, GroSS is the first method to simultaneously train differing numbers of groups within a single layer, as well as all possible combinations between layers. In doing so, GroSS trains an entire grouped convolution architecture search-space concurrently. We demonstrate this with a proof-of-concept exhaustive architecure search with a performance objective. GroSS represents a significant step towards liberating network architecture search from the burden of training and finetuning.
Summary
---
This paper proposes to learn simultaneously all the parameters of grouped convolutions by factorizing the weights of the convolutions as a sum of lower rank tensors. This enables architecture search for the convolution parameters in a differentiable and efficient way.
Comments
---
I think was paper is well written, and was clear at least until 3.2. I believe some clarifications could be useful here, it is not written clearly that t' and u', the 2 first dimensions of the core are R times smaller than t and u. There is some explanation in the bracket (3) but 1) it should be stated clearly in the text, 2) I believe there are several typos on the 4th line of bracket (3) making it hard to understand.
I did not know about the expansion function, and while I trust the authors that it is correctly used, I would have like either more explanations on how it works or some reference.
Can you justify the softmax and the very high temperature? For N = 8, s_1 will be sampled 98.2% of the time s_2 1.8% and the other sampling probabilities are close to neglibigle. While I understand it seems to work better in practice, it looks extremely aggressive.
In 4.4 you say you perform finetuning for 150 epochs, which is huge, while on the abstract you said "GroSS represents a significant step towards
liberating network architecture search from the burden of training and finetuning". Can you comment?
As you say GroSS is an alternative to NAS (for the convolutions parameters that is), is the GroSS method proposed really faster and more accurate than a NAS baseline for finding these architectures?
I don't find the column titles in Table 3 to be always informative. "After train" means after the finetuning? I took me some time to realize the delta was the delta in accuracies, it is not very informative and it was not clear for me for some time what it meant. Either the titles should be chosen more carefully or the caption should be more precise I believe.
In figure 1, the legend should be more informative, at least incorporate a "alpha" or "temperature" title in the legend.
Conclusion
---
While the method is interesting I am wondering whether GroSS enables more efficient architecture search that tradional methods as there is still a long finetuning step, furthermore it can only be applied to grouped convolutions parameters. As the authors present it in the abstract and introduction as an alternative to NAS, I believe a comparison to a NAS would be needed.