We propose a novel training approach for improving the learning of generalizing features in neural networks. We augment the network with a reverse pass which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, i.e. racecar training, leads to significantly more general features to be extracted from a given data set. We demonstrate in our paper that a network obtained in this way is continually trained for the original task, it outperforms baseline models trained in a regular fashion. This improved performance is visible for a wide range of learning tasks from classification, to regression and stylization. In addition, networks trained with our approach exhibit improved performance for task transfers. We additionally analyze the mutual information of our networks to explain the improved generalizing capabilities.
This paper attempts to learn general and reusable features for transfer learning tasks. The authors propose a training paradigm called Racecar Training. The core idea of it is to operate a reverse pass for the network. The authors use mutual information to analyze the network for its improved generalizing capabilities. They also conduct experiments on classification, regression and stylization to validate their method’s effectiveness.
Pros:
The reverse pass idea is similar to auto-encoder paradigm that discards redundant information and only save the essential low dimensional one by comparing the original data and the decoding data. The general feature the authors mentioned is like the essential low dimensional information, which is reasonable.
Cons:
1.	I think the main drawbacks of this paper is that the authors make a poor presentation. The authors talk about learning general features with which the model can use on new task in the title, introduction and even the whole paper. However, there are no explicit general features learned during the learning procedure. They only perform the reverse pass when learning in the original task. Even the structure of networks does not change at all. The general and reusable feature is only an explanation of the improved performance. I think the authors should change this explanation to a more convincing one. For example, the network may learn general and reusable weights or parameters since it is the model learned that will be applied to new tasks instead of the features. 
2.	The analysis by mutual information makes the paper hard to follow. The figures such as figure 2 are so confusing. There are so many points and lines in each picture. What do they mean? What are the x-axis and y-axis? The authors also do not explain what the meaning of different mutual information are. 
3.	The symbols are chaotic. The authors explain “RR^3” means n=1 in equation 1. Then what does n equal to in RR^1? The authors explain “AB” means the model was trained for task A during phase I, and is then trained for task B as transfer in phase II. Then what does AA/AB in “Std_{AA/AB}” mean? 
4.	In the last paragraph of page 3, the authors mention orange color. However, there are only green, blue and yellow color in Figure 1.
5.	In experiments, I do not see obvious advantage of the proposed method. For example, in Figure 6, the test accuracy of Std_{AA} is 0.9842 while that of RR^3_{AA} is only 0.9859. In Figure 7, the test accuracy of Std_{AA} is 0.8377 while that of RR^13_{AA} is 0.8711. The transfer tasks (referred as AB) also show minor advantages. With minor advantages but increased requirements for memory and additional computations (e.g., 61.13% slower per epoch for the MNIST tests mentioned by the authors), the proposed method shows very limited values.