Winner-Take-All Autoencoders
In this paper, we propose a winner-take-all method for learning hierarchical sparse representations in an unsupervised fashion. We first introduce fully-connected winner-take-all autoencoders which use mini-batch statistics to directly enforce a lifetime sparsity in the activations of the hidden units. We then propose the convolutional winner-take-all autoencoder which combines the benefits of convolutional architectures and autoencoders for learning shift-invariant sparse representations. We describe a way to train convolutional autoencoders layer by layer, where in addition to lifetime sparsity, a spatial sparsity within each feature map is achieved using winner-take-all activation functions. We will show that winner-take-all autoencoders can be used to to learn deep sparse representations from the MNIST, CIFAR-10, ImageNet, Street View House Numbers and Toronto Face datasets, and achieve competitive classification performance.
The authors present a fully connected auto-encoder with a new sparsity constraint called the lifetime sparsity. For each hidden unit across the mini-batch, they rank the activation values, keeping only the top-k% for reconstruction. The approach is appealing because they don't need to find a hard threshold and it makes sure every hidden unit/filter is updated (no dead filters because their activation was below the threshold).Their encoder is a deep stack of ReLu and the decoder is shallow and linear (note that usually non-symmetric auto-encoders lead to worse results). They also show how to apply to RBM. The effect of sparsity is very effective and noticeable on the images depicting the filters.They extend this auto-encoder in a convolutional/deconvolutional framework, making it possible to train on larger images than MNIST or TFD. They add a spatial sparsity, keeping the top activation per feature map for the reconstruction and combine it with the lifetime sparsity presented before.Having worked on auto-encoders in the past, I really like the paper and I think this will be very interesting to many people in this field (researchers in auto-encoder representation learning). I have a few questions/requests:* can you present the reconstruction of a full image on image net using your architecture? i.e. presenting the original image on one side and the reconstructed version on the side. * you only evaluate with an SVM on top of the representation. Does the pre-initialization improves the performance of the same architecture but randomly initialized if you fine-tune the model? can you provide results in that scenario?minor comments L52 "no hyper-parameter to be tuned" what about the initialization technique you choose? the learning rate? the number of hidden units? the activation function? I would consider those as hyper-parameters. L230 "it requires all the dictionary atoms to contribute in the reconstruction" if I look at Fig 5f, there are a few feature maps that are totally black (meaning no activation at all, or very close to 0). Will the corresponding atoms contribute to the reconstruction? same idea at L236, "every filter will get updated", maybe precise that the filter needs to be activated. Maybe you can measure during training which proportion of the filters are not activated. L295 can you point to the url if the paper gets accepted? thanks. Table 2b, CONV-WTA = Stacked CONV-WTA 128 & 2048 maps? precise in the caption. Fig 4a, filters look very weird, can you explain?precise how ranking at each iteration affect the training time (should be very negligible but useful to state that it does compare to normal auto encoder training)