We apply multi-task learning to image classification tasks on MNIST-like datasets. MNIST dataset has been referred to as the drosophila of machine learning and has been the testbed of many learning theories. The NotMNIST dataset and the FashionMNIST dataset have been created with the MNIST dataset as reference. In this work, we exploit these MNIST-like datasets for multi-task learning. The datasets are pooled together for learning the parameters of joint classification networks. Then the learned parameters are used as the initial parameters to retrain disjoint classification networks. The baseline recognition model are all-convolution neural networks. Without multi-task learning, the recognition accuracies for MNIST, NotMNIST and FashionMNIST are 99.56%, 97.22% and 94.32% respectively. With multi-task learning to pre-train the networks, the recognition accuracies are respectively 99.70%, 97.46% and 95.25%. The results re-affirm that multi-task learning framework, even with data with different genres, does lead to significant improvement.
The manuscript mainly utilizing the data from all three MNIST-like datasets to pre-train the parameters of joint classification networks, and the pre-trained parameters are utilized to initialize the disjoint classification networks (of the three datasets).
The presented idea is quite simple and the authors only re-affirm that multi-task learning can lead to performance improvement by simultaneously leverage the information of multiple tasks. There is no technique contribution.
Pros:
1.	The main idea is clearly presented.
2.	It is interesting to visualize the results obtained with/without multi-task learning in Figure 6.
Cons:
1.	The contribution is quite limited since the authors only apply multi-task learning to the three MNIST-like datasets and there is no technique contribution.
2.	There is no difference between the architecture of the single-task learning network and multi-task learning network.
3.	Many unclear points, e.g., there is no description for “zero-padding” and why it can enhance target label. What is the “two-stage learning rate decay scheme” and why it is implemented? It is also unclear what can we observed from Figure 4.