Dynamic Filter Networks ESAT-PSI, KU Leuven, iMinds Tinne Tuytelaars ESAT-PSI, KU Leuven, iMinds
In a traditional convolutional layer, the learned filters stay fixed after training. In contrast, we introduce a new framework, the Dynamic Filter Network, where filters are generated dynamically conditioned on an input. We show that this architecture is a powerful one, with increased flexibility thanks to its adaptive nature, yet without an excessive increase in the number of model parameters. A wide variety of filtering operations can be learned this way, including local spatial transformations, but also others like selective (de)blurring or adaptive feature extraction. Moreover, multiple such layers can be combined, e.g. in a recurrent architecture. We demonstrate the effectiveness of the dynamic filter network on the tasks of video and stereo prediction, and reach state-of-the-art performance on the moving MNIST dataset with a much smaller model. By visualizing the learned filters, we illustrate that the network has picked up flow information by only looking at unlabelled training data. This suggests that the network can be used to pretrain networks for various supervised tasks in an unsupervised way, like optical flow and depth estimation.
This paper introduces a new framework termed "Dynamic Filter Networks", where the filters of a convolutional layer are not fixed but are predicted from a small subnetwork. The experiments contain a 1) sanity check (or a unit test) of the proposed approach, and networks that 2) take 10 frames of moving MNIST digits from [19] and predict the next 10 frames, 3) that predict 2 future frames in a highway driving application, and 4) that predict stereo views, posed as generating right view given left. In particular the experiments on moving MNIST show that the model can perform this task better than previous work and with many fewer parameters than previous work [18,19].
First, I'd like to say that this paper is very nicely written and was easy to read and understand. I also appreciated the thoroughness of experiments showing results in 1) unit test, 2) MNIST toy data, and 3) a slightly more real application.  My major concern with this paper is with marketing the dynamic filter network as something entirely new. Certainly, the model predicts some parameters that happen to be used in a convolution, but that simply means that you have a slightly more complex mapping from inputs to outputs. The approach is still entirely parametric, you're merely tweaking the functional form of the mapping and allowing your network slightly more complex interactions. The authors devote a paragraph to this point on line 114, stating that:  ``` we make a distinction between *model parameters* and *dynamically generated parameters*. Model parameters denote the layer parameters that are initialized in advance and only updated during training. They are the same for all samples. Dynamically generated parameters are sample-specific, and are produced on-the-fly without a need for initialization. ```  this is incorrect. The dynamic filter network is exactly identical to every other normal network: it still has a fixed set of parameters that are initialized in advance and updated during training. You're merely designing a more complex mapping. The definition of a parameter is not "the `weight` field in a convolutional layer". It is merely a number that changes during backprop. The authors spend an entire section 3.3 on drawing analogies to STNs, ResNets, etc. and I find these very far fetched and abstract comparisons.  Finally, the experiments are nicely done and I appreciate Figure 7 that visualizes what the network learns.  In summary, I think the authors make a big deal out of a simple idea, market it in a strange way, but ultimately do a nice job in execution: writing, experiments, and presenting the results. I think the experiments in this paper nicely highlight the strength of exactly this kind of model, where predicting the filters of a conv layer dynamically makes a lot of sense architecturally. Overall I like the paper and I'd be happy to see it accepted.   Minor: - "Humans are good at predicting another view from related views". Are they? This is a poor way to start the intro of this paper. In general I'm not a fan of the first paragraph. - The authors have a strange interpretation of the Softmax function, claiming a few times that it helps "encourage elements to only have a few non-zero elements" (line 248). This is by no means the case. The network can choose to have high magnitude vectors where Softmax returns very peaky results, but it could also in principle return very diffuse vectors.